Oct 30 21:49:08.731689 ip-10-0-128-245 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 21:49:09.243833 ip-10-0-128-245 kubenswrapper[2573]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 21:49:09.243833 ip-10-0-128-245 kubenswrapper[2573]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 21:49:09.243833 ip-10-0-128-245 kubenswrapper[2573]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 21:49:09.244373 ip-10-0-128-245 kubenswrapper[2573]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 21:49:09.248814 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.248739    2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:49:09.248814 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.248813    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:49:09.248881 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.248819    2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:49:09.248881 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.248822    2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:49:09.248881 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.248826    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:49:09.248881 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.248830    2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:49:09.248881 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.248835    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:49:09.248881 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.248838    2573 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:49:09.248881 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.248841    2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:49:09.248881 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.248844    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:49:09.248881 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.248847    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:49:09.248881 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.248849    2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:49:09.248881 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.248852    2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:49:09.248881 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.248854    2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:49:09.248881 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.248857    2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:49:09.248881 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.248860    2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:49:09.248881 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.248862    2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:49:09.248881 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.248865    2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:49:09.248881 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.248871    2573 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:49:09.248881 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.248874    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:49:09.248881 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.248877    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:49:09.249321 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.248880    2573 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:49:09.249321 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.248882    2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:49:09.249321 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.248885    2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:49:09.249321 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.248889    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:49:09.249321 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.248892    2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:49:09.249321 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.248895    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:49:09.249321 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.248897    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:49:09.249321 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.248900    2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:49:09.249321 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.248903    2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:49:09.249321 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.248905    2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:49:09.249321 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.248907    2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:49:09.249321 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.248910    2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:49:09.249321 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.248913    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:49:09.249321 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.248915    2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:49:09.249321 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.248918    2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:49:09.249321 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.248920    2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:49:09.249321 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.248923    2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:49:09.249321 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.248925    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:49:09.249321 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.248934    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:49:09.249321 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.248936    2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:49:09.249829 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.248939    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:49:09.249829 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.248942    2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:49:09.249829 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.248944    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:49:09.249829 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.248947    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:49:09.249829 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.248949    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:49:09.249829 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.248952    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:49:09.249829 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.248955    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:49:09.249829 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.248957    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:49:09.249829 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.248960    2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:49:09.249829 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.248963    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:49:09.249829 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.248965    2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:49:09.249829 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.248967    2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:49:09.249829 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.248970    2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:49:09.249829 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.248974    2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:49:09.249829 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.248976    2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:49:09.249829 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.248979    2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:49:09.249829 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.248981    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:49:09.249829 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.248984    2573 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:49:09.249829 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.248986    2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:49:09.250282 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.248989    2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:49:09.250282 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.248992    2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:49:09.250282 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.248994    2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:49:09.250282 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.248997    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:49:09.250282 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.248999    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:49:09.250282 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.249002    2573 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:49:09.250282 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.249004    2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:49:09.250282 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.249007    2573 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:49:09.250282 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.249009    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:49:09.250282 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.249012    2573 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:49:09.250282 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.249014    2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:49:09.250282 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.249017    2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:49:09.250282 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.249025    2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:49:09.250282 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.249028    2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:49:09.250282 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.249031    2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:49:09.250282 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.249033    2573 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:49:09.250282 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.249035    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:49:09.250282 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.249038    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:49:09.250282 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.249040    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:49:09.250282 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.249043    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:49:09.250761 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.249045    2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:49:09.250761 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.249048    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:49:09.250761 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.249050    2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:49:09.250761 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.249052    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:49:09.250761 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.249055    2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:49:09.250761 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.249058    2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:49:09.250761 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.249061    2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:49:09.250761 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.249063    2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:49:09.250761 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.249066    2573 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:49:09.250761 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.249069    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:49:09.250761 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.249071    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:49:09.250761 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.249074    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:49:09.250761 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.249076    2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:49:09.250761 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.249078    2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:49:09.250761 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.249081    2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:49:09.250761 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.249084    2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:49:09.250761 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.249086    2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:49:09.250761 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.249088    2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:49:09.250761 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.249091    2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:49:09.250761 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.249093    2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:49:09.251229 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.249096    2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:49:09.251229 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.249098    2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:49:09.251229 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.249102    2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:49:09.251229 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.249106    2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:49:09.251229 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.249110    2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:49:09.251229 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.249120    2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:49:09.251229 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.250092    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:49:09.251229 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.250100    2573 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:49:09.251229 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.250103    2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:49:09.251229 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.250107    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:49:09.251229 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.250111    2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:49:09.251229 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.250114    2573 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:49:09.251229 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.250121    2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:49:09.251229 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.250126    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:49:09.251229 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.250129    2573 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:49:09.251229 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.250131    2573 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:49:09.251229 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.250134    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:49:09.251229 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.250138    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:49:09.251229 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.250141    2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:49:09.251701 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.250143    2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:49:09.251701 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.250146    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:49:09.251701 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.250149    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:49:09.251701 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.250152    2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:49:09.251701 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.250154    2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:49:09.251701 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.250157    2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:49:09.251701 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.250160    2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:49:09.251701 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.250162    2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:49:09.251701 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.250165    2573 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:49:09.251701 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.250167    2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:49:09.251701 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.250170    2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:49:09.251701 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.250172    2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:49:09.251701 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.250175    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:49:09.251701 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.250178    2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:49:09.251701 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.250180    2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:49:09.251701 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.250183    2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:49:09.251701 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.250185    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:49:09.251701 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.250188    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:49:09.251701 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.250190    2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:49:09.251701 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.250200    2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:49:09.252194 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.250203    2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:49:09.252194 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.250206    2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:49:09.252194 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.250209    2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:49:09.252194 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.250211    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:49:09.252194 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.250214    2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:49:09.252194 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.250216    2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:49:09.252194 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.250219    2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:49:09.252194 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.250221    2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:49:09.252194 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.250224    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:49:09.252194 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.250226    2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:49:09.252194 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.250229    2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:49:09.252194 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.250232    2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:49:09.252194 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.250234    2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:49:09.252194 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.250237    2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:49:09.252194 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.250239    2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:49:09.252194 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.250242    2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:49:09.252194 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.250244    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:49:09.252194 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.250247    2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:49:09.252640 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.250251    2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:49:09.252640 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.250255    2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:49:09.252640 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.250258    2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:49:09.252640 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.250260    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:49:09.252640 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.250263    2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:49:09.252640 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.250266    2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:49:09.252640 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.250268    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:49:09.252640 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.250271    2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:49:09.252640 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.250273    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:49:09.252640 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.250276    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:49:09.252640 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.250279    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:49:09.252640 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.250282    2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:49:09.252640 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.250284    2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:49:09.252640 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.250287    2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:49:09.252640 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.250296    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:49:09.252640 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.250299    2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:49:09.252640 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.250302    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:49:09.252640 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.250305    2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:49:09.252640 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.250307    2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:49:09.253098 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.250310    2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:49:09.253098 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.250313    2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:49:09.253098 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.250315    2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:49:09.253098 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.250318    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:49:09.253098 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.250320    2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:49:09.253098 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.250323    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:49:09.253098 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.250326    2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:49:09.253098 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.250328    2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:49:09.253098 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.250331    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:49:09.253098 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.250333    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:49:09.253098 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.250336    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:49:09.253098 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.250339    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:49:09.253098 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.250341    2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:49:09.253098 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.250344    2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:49:09.253098 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.250347    2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:49:09.253098 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.250350    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:49:09.253098 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.250353    2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:49:09.253098 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.250355    2573 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:49:09.253098 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.250358    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:49:09.253098 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.250360    2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:49:09.253596 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.250363    2573 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:49:09.253596 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.250366    2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:49:09.253596 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.250372    2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:49:09.253596 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.250375    2573 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:49:09.253596 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.250377    2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:49:09.253596 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.250380    2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:49:09.253596 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.250382    2573 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:49:09.253596 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.250384    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:49:09.253596 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.250393    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:49:09.253596 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.250395    2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:49:09.253596 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.250410    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:49:09.253596 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.250413    2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:49:09.253596 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.250416    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:49:09.253596 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.250418    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:49:09.253596 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.250421    2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:49:09.253596 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.250423    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:49:09.253596 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.251028    2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:49:09.253596 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.251033    2573 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:49:09.253596 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.251036    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:49:09.253596 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.251039    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:49:09.254359 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.251041    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:49:09.254359 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.251044    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:49:09.254359 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.251046    2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:49:09.254359 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.251049    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:49:09.254359 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.251051    2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:49:09.254359 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.251054    2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:49:09.254359 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.251057    2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:49:09.254359 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.251059    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:49:09.254359 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.251062    2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:49:09.254359 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.251065    2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:49:09.254359 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.251068    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:49:09.254359 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.251070    2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:49:09.254359 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.251074    2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:49:09.254359 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.251077    2573 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:49:09.254359 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.251083    2573 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:49:09.254359 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.251087    2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:49:09.254359 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.251091    2573 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:49:09.254359 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.251094    2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:49:09.254359 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.251096    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:49:09.254941 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.251102    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:49:09.254941 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.251111    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:49:09.254941 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.251114    2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:49:09.254941 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.251116    2573 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:49:09.254941 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.251119    2573 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:49:09.254941 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.251121    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:49:09.254941 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.251124    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:49:09.254941 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.251126    2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:49:09.254941 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.251129    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:49:09.254941 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.251131    2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:49:09.254941 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.251135    2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:49:09.254941 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.251137    2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:49:09.254941 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.251140    2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:49:09.254941 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.251142    2573 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:49:09.254941 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.251145    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:49:09.254941 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.251147    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:49:09.254941 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.251150    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:49:09.254941 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.251152    2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:49:09.254941 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.251155    2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:49:09.254941 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.251157    2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:49:09.255437 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.251160    2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:49:09.255437 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.251162    2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:49:09.255437 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.251165    2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:49:09.255437 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.251167    2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:49:09.255437 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.251170    2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:49:09.255437 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.251173    2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:49:09.255437 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.251175    2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:49:09.255437 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.251179    2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:49:09.255437 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.251181    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:49:09.255437 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.251184    2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:49:09.255437 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.251186    2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:49:09.255437 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.251189    2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:49:09.255437 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.251191    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:49:09.255437 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.251194    2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:49:09.255437 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.251205    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:49:09.255437 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.251208    2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:49:09.255437 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.251210    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:49:09.255437 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.251213    2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:49:09.255437 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.251215    2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:49:09.255956 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.251218    2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:49:09.255956 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.251221    2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:49:09.255956 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.251223    2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:49:09.255956 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.251226    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:49:09.255956 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.251228    2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:49:09.255956 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.251231    2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:49:09.255956 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.251234    2573 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:49:09.255956 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.251236    2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:49:09.255956 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.251239    2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:49:09.255956 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.251241    2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:49:09.255956 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.251244    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:49:09.255956 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.251247    2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:49:09.255956 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.251249    2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:49:09.255956 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.251252    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:49:09.255956 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.251255    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:49:09.255956 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.251257    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:49:09.255956 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.251260    2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:49:09.255956 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.251262    2573 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:49:09.255956 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.251264    2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:49:09.256394 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.251267    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:49:09.256394 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.251270    2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:49:09.256394 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.251273    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:49:09.256394 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.251275    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:49:09.256394 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.251278    2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:49:09.256394 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.251280    2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:49:09.256394 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.251283    2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:49:09.256394 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.251285    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:49:09.256394 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.251288    2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:49:09.256394 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.251296    2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:49:09.256394 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.251298    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:49:09.256394 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.251301    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:49:09.256394 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.251303    2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:49:09.256394 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.251306    2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:49:09.256394 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.251308    2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:49:09.256394 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.251311    2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:49:09.256394 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.251313    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:49:09.256394 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.251316    2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:49:09.256394 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.251318    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:49:09.256394 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.251320    2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:49:09.256887 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.251323    2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:49:09.256887 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.251326    2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:49:09.256887 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.251328    2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:49:09.256887 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.251331    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:49:09.256887 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.251334    2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:49:09.256887 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251433    2573 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 21:49:09.256887 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251443    2573 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 21:49:09.256887 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251452    2573 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 21:49:09.256887 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251457    2573 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 21:49:09.256887 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251461    2573 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 21:49:09.256887 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251465    2573 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 21:49:09.256887 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251469    2573 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 21:49:09.256887 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251473    2573 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 21:49:09.256887 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251477    2573 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 21:49:09.256887 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251479    2573 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 21:49:09.256887 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251483    2573 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 21:49:09.256887 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251486    2573 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 21:49:09.256887 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251489    2573 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 21:49:09.256887 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251492    2573 flags.go:64] FLAG: --cgroup-root=""
Oct 30 21:49:09.256887 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251495    2573 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 21:49:09.256887 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251498    2573 flags.go:64] FLAG: --client-ca-file=""
Oct 30 21:49:09.256887 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251501    2573 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 21:49:09.256887 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251509    2573 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 21:49:09.257434 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251515    2573 flags.go:64] FLAG: --cluster-domain=""
Oct 30 21:49:09.257434 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251518    2573 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 21:49:09.257434 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251521    2573 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 21:49:09.257434 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251524    2573 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 21:49:09.257434 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251528    2573 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 21:49:09.257434 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251532    2573 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 21:49:09.257434 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251535    2573 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 21:49:09.257434 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251538    2573 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 21:49:09.257434 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251541    2573 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 21:49:09.257434 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251544    2573 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 21:49:09.257434 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251547    2573 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 21:49:09.257434 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251550    2573 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 21:49:09.257434 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251553    2573 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 21:49:09.257434 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251556    2573 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 21:49:09.257434 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251560    2573 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 21:49:09.257434 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251567    2573 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 21:49:09.257434 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251570    2573 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 21:49:09.257434 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251573    2573 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 21:49:09.257434 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251576    2573 flags.go:64] FLAG: --enable-server="true"
Oct 30 21:49:09.257434 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251578    2573 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 21:49:09.257434 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251585    2573 flags.go:64] FLAG: --event-burst="100"
Oct 30 21:49:09.257434 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251588    2573 flags.go:64] FLAG: --event-qps="50"
Oct 30 21:49:09.257434 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251591    2573 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 21:49:09.257434 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251594    2573 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 21:49:09.258055 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251597    2573 flags.go:64] FLAG: --eviction-hard=""
Oct 30 21:49:09.258055 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251601    2573 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 21:49:09.258055 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251604    2573 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 21:49:09.258055 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251607    2573 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 21:49:09.258055 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251610    2573 flags.go:64] FLAG: --eviction-soft=""
Oct 30 21:49:09.258055 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251613    2573 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 21:49:09.258055 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251616    2573 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 21:49:09.258055 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251619    2573 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 21:49:09.258055 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251627    2573 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 21:49:09.258055 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251630    2573 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 21:49:09.258055 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251633    2573 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 21:49:09.258055 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251636    2573 flags.go:64] FLAG: --feature-gates=""
Oct 30 21:49:09.258055 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251640    2573 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 21:49:09.258055 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251643    2573 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 21:49:09.258055 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251647    2573 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 21:49:09.258055 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251650    2573 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 21:49:09.258055 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251653    2573 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 21:49:09.258055 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251656    2573 flags.go:64] FLAG: --help="false"
Oct 30 21:49:09.258055 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251659    2573 flags.go:64] FLAG: --hostname-override="ip-10-0-128-245.us-west-2.compute.internal"
Oct 30 21:49:09.258055 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251662    2573 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 21:49:09.258055 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251665    2573 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 21:49:09.258055 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251667    2573 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 21:49:09.258055 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251671    2573 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 21:49:09.258593 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251676    2573 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 21:49:09.258593 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251679    2573 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 21:49:09.258593 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251681    2573 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 21:49:09.258593 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251684    2573 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 21:49:09.258593 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251687    2573 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 21:49:09.258593 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251690    2573 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 21:49:09.258593 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251693    2573 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 21:49:09.258593 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251696    2573 flags.go:64] FLAG: --kube-reserved=""
Oct 30 21:49:09.258593 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251698    2573 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 21:49:09.258593 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251701    2573 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 21:49:09.258593 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251704    2573 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 21:49:09.258593 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251707    2573 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 21:49:09.258593 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251710    2573 flags.go:64] FLAG: --lock-file=""
Oct 30 21:49:09.258593 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251713    2573 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 21:49:09.258593 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251716    2573 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 21:49:09.258593 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251719    2573 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 21:49:09.258593 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251725    2573 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 21:49:09.258593 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251727    2573 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 21:49:09.258593 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251735    2573 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 21:49:09.258593 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251739    2573 flags.go:64] FLAG: --logging-format="text"
Oct 30 21:49:09.258593 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251742    2573 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 21:49:09.258593 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251745    2573 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 21:49:09.258593 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251748    2573 flags.go:64] FLAG: --manifest-url=""
Oct 30 21:49:09.258593 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251752    2573 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 21:49:09.258593 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251757    2573 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 21:49:09.259193 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251760    2573 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 21:49:09.259193 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251763    2573 flags.go:64] FLAG: --max-pods="110"
Oct 30 21:49:09.259193 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251766    2573 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 21:49:09.259193 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251769    2573 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 21:49:09.259193 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251772    2573 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 21:49:09.259193 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251775    2573 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 21:49:09.259193 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251778    2573 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 21:49:09.259193 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251782    2573 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 21:49:09.259193 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251785    2573 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 21:49:09.259193 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251792    2573 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 21:49:09.259193 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251795    2573 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 21:49:09.259193 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251798    2573 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 21:49:09.259193 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251801    2573 flags.go:64] FLAG: --pod-cidr=""
Oct 30 21:49:09.259193 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251804    2573 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 21:49:09.259193 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251807    2573 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 21:49:09.259193 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251810    2573 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 21:49:09.259193 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251813    2573 flags.go:64] FLAG: --port="10250"
Oct 30 21:49:09.259193 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251816    2573 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 21:49:09.259193 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251818    2573 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0f5152ebb45a0e416"
Oct 30 21:49:09.259193 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251821    2573 flags.go:64] FLAG: --qos-reserved=""
Oct 30 21:49:09.259193 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251824    2573 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 21:49:09.259193 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251827    2573 flags.go:64] FLAG: --register-node="true"
Oct 30 21:49:09.259193 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251830    2573 flags.go:64] FLAG: --register-with-taints=""
Oct 30 21:49:09.259193 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251834    2573 flags.go:64] FLAG: --registry-burst="10"
Oct 30 21:49:09.259193 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251837    2573 flags.go:64] FLAG: --registry-qps="5"
Oct 30 21:49:09.259193 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251839    2573 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 21:49:09.259851 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251849    2573 flags.go:64] FLAG: --reserved-memory=""
Oct 30 21:49:09.259851 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251853    2573 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 21:49:09.259851 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251856    2573 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 21:49:09.259851 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251859    2573 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 21:49:09.259851 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251864    2573 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 21:49:09.259851 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251867    2573 flags.go:64] FLAG: --runonce="false"
Oct 30 21:49:09.259851 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251870    2573 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 21:49:09.259851 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251873    2573 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 21:49:09.259851 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251876    2573 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 21:49:09.259851 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251879    2573 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 21:49:09.259851 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251882    2573 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 21:49:09.259851 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251885    2573 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 21:49:09.259851 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251888    2573 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 21:49:09.259851 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251893    2573 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 21:49:09.259851 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251896    2573 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 21:49:09.259851 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251899    2573 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 21:49:09.259851 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251902    2573 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 21:49:09.259851 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251905    2573 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 21:49:09.259851 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251907    2573 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 21:49:09.259851 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251910    2573 flags.go:64] FLAG: --system-cgroups=""
Oct 30 21:49:09.259851 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251913    2573 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 21:49:09.259851 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251919    2573 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 21:49:09.259851 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251921    2573 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 21:49:09.259851 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251924    2573 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 21:49:09.259851 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251931    2573 flags.go:64] FLAG: --tls-min-version=""
Oct 30 21:49:09.260461 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251934    2573 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 21:49:09.260461 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251936    2573 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 21:49:09.260461 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251939    2573 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 21:49:09.260461 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251942    2573 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 21:49:09.260461 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251945    2573 flags.go:64] FLAG: --v="2"
Oct 30 21:49:09.260461 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251950    2573 flags.go:64] FLAG: --version="false"
Oct 30 21:49:09.260461 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251954    2573 flags.go:64] FLAG: --vmodule=""
Oct 30 21:49:09.260461 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251958    2573 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 21:49:09.260461 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.251967    2573 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 21:49:09.260461 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252158    2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:49:09.260461 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252162    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:49:09.260461 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252167    2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:49:09.260461 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252170    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:49:09.260461 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252173    2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:49:09.260461 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252175    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:49:09.260461 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252178    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:49:09.260461 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252180    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:49:09.260461 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252183    2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:49:09.260461 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252186    2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:49:09.260461 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252188    2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:49:09.260461 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252192    2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:49:09.260461 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252195    2573 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:49:09.260997 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252198    2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:49:09.260997 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252200    2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:49:09.260997 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252204    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:49:09.260997 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252206    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:49:09.260997 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252209    2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:49:09.260997 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252212    2573 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:49:09.260997 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252214    2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:49:09.260997 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252217    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:49:09.260997 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252219    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:49:09.260997 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252222    2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:49:09.260997 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252224    2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:49:09.260997 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252228    2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:49:09.260997 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252232    2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:49:09.260997 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252234    2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:49:09.260997 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252237    2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:49:09.260997 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252239    2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:49:09.260997 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252242    2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:49:09.260997 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252244    2573 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:49:09.260997 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252254    2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:49:09.261480 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252264    2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:49:09.261480 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252267    2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:49:09.261480 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252272    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:49:09.261480 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252274    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:49:09.261480 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252277    2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:49:09.261480 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252279    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:49:09.261480 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252282    2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:49:09.261480 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252285    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:49:09.261480 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252288    2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:49:09.261480 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252290    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:49:09.261480 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252293    2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:49:09.261480 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252296    2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:49:09.261480 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252299    2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:49:09.261480 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252301    2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:49:09.261480 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252304    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:49:09.261480 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252306    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:49:09.261480 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252308    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:49:09.261480 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252311    2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:49:09.261480 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252314    2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:49:09.261930 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252316    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:49:09.261930 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252318    2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:49:09.261930 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252321    2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:49:09.261930 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252323    2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:49:09.261930 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252326    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:49:09.261930 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252328    2573 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:49:09.261930 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252331    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:49:09.261930 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252333    2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:49:09.261930 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252336    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:49:09.261930 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252338    2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:49:09.261930 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252341    2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:49:09.261930 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252344    2573 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:49:09.261930 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252346    2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:49:09.261930 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252349    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:49:09.261930 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252357    2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:49:09.261930 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252361    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:49:09.261930 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252364    2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:49:09.261930 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252366    2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:49:09.261930 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252369    2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:49:09.262380 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252372    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:49:09.262380 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252374    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:49:09.262380 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252377    2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:49:09.262380 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252379    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:49:09.262380 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252382    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:49:09.262380 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252384    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:49:09.262380 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252387    2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:49:09.262380 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252389    2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:49:09.262380 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252392    2573 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:49:09.262380 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252394    2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:49:09.262380 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252408    2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:49:09.262380 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252411    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:49:09.262380 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252414    2573 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:49:09.262380 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252416    2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:49:09.262380 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252419    2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:49:09.262380 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252421    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:49:09.262380 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252424    2573 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:49:09.262380 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252427    2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:49:09.262380 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252429    2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:49:09.262380 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252432    2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:49:09.262870 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252434    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:49:09.262870 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252437    2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:49:09.262870 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252439    2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:49:09.262870 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252442    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:49:09.262870 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252444    2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:49:09.262870 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252447    2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:49:09.262870 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252449    2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:49:09.262870 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252452    2573 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:49:09.262870 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252461    2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:49:09.262870 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252464    2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:49:09.262870 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252467    2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:49:09.262870 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252469    2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:49:09.262870 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252472    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:49:09.262870 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252474    2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:49:09.262870 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252477    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:49:09.262870 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252479    2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:49:09.263248 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.252506    2573 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 21:49:09.263248 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252556    2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:49:09.263248 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252561    2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:49:09.263248 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252564    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:49:09.263248 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252568    2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:49:09.263248 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252570    2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:49:09.263248 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252573    2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:49:09.263248 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252575    2573 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:49:09.263248 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252578    2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:49:09.263248 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252581    2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:49:09.263248 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252584    2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:49:09.263248 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252586    2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:49:09.263248 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252589    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:49:09.263248 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252592    2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:49:09.263248 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252594    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:49:09.263248 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252597    2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:49:09.263637 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252599    2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:49:09.263637 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252602    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:49:09.263637 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252604    2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:49:09.263637 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252607    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:49:09.263637 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252609    2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:49:09.263637 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252612    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:49:09.263637 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252614    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:49:09.263637 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252617    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:49:09.263637 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252619    2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:49:09.263637 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252629    2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:49:09.263637 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252632    2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:49:09.263637 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252635    2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:49:09.263637 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252637    2573 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:49:09.263637 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252640    2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:49:09.263637 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252643    2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:49:09.263637 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252645    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:49:09.263637 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252649    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:49:09.263637 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252651    2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:49:09.263637 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252654    2573 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:49:09.263637 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252656    2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:49:09.264109 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252659    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:49:09.264109 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252661    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:49:09.264109 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252664    2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:49:09.264109 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252666    2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:49:09.264109 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252670    2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:49:09.264109 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252674    2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:49:09.264109 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252677    2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:49:09.264109 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252680    2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:49:09.264109 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252683    2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:49:09.264109 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252685    2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:49:09.264109 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252688    2573 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:49:09.264109 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252692    2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:49:09.264109 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252696    2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:49:09.264109 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252699    2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:49:09.264109 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252702    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:49:09.264109 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252704    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:49:09.264109 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252707    2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:49:09.264109 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252709    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:49:09.264555 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252712    2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:49:09.264555 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252723    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:49:09.264555 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252725    2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:49:09.264555 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252729    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:49:09.264555 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252737    2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:49:09.264555 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252740    2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:49:09.264555 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252743    2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:49:09.264555 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252745    2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:49:09.264555 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252748    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:49:09.264555 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252750    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:49:09.264555 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252754    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:49:09.264555 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252757    2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:49:09.264555 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252759    2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:49:09.264555 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252761    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:49:09.264555 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252764    2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:49:09.264555 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252766    2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:49:09.264555 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252769    2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:49:09.264555 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252772    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:49:09.264555 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252774    2573 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:49:09.264555 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252776    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:49:09.265015 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252779    2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:49:09.265015 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252781    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:49:09.265015 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252784    2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:49:09.265015 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252786    2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:49:09.265015 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252789    2573 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:49:09.265015 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252791    2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:49:09.265015 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252794    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:49:09.265015 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252796    2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:49:09.265015 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252799    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:49:09.265015 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252801    2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:49:09.265015 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252804    2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:49:09.265015 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252806    2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:49:09.265015 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252809    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:49:09.265015 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252812    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:49:09.265015 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252814    2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:49:09.265015 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252818    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:49:09.265015 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252820    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:49:09.265015 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252828    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:49:09.265015 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252831    2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:49:09.265486 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252833    2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:49:09.265486 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252836    2573 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:49:09.265486 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252839    2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:49:09.265486 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252842    2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:49:09.265486 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252846    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:49:09.265486 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252849    2573 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:49:09.265486 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252851    2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:49:09.265486 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252854    2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:49:09.265486 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252856    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:49:09.265486 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252859    2573 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:49:09.265486 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252862    2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:49:09.265486 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252864    2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:49:09.265486 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252866    2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:49:09.265486 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.252869    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:49:09.265486 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.261072    2573 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 21:49:09.265486 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.261087    2573 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 21:49:09.265486 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.261992    2573 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 21:49:09.265486 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.261999    2573 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 21:49:09.265486 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262060    2573 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:49:09.265486 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262066    2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:49:09.265990 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262071    2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:49:09.265990 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262074    2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:49:09.265990 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262077    2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:49:09.265990 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262080    2573 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:49:09.265990 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262083    2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:49:09.265990 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262085    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:49:09.265990 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262088    2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:49:09.265990 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262091    2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:49:09.265990 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262094    2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:49:09.265990 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262096    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:49:09.265990 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262099    2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:49:09.265990 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262102    2573 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:49:09.265990 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262104    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:49:09.265990 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262107    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:49:09.265990 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262110    2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:49:09.265990 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262112    2573 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:49:09.265990 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262115    2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:49:09.265990 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262118    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:49:09.265990 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262120    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:49:09.265990 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262123    2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:49:09.266478 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262126    2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:49:09.266478 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262129    2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:49:09.266478 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262132    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:49:09.266478 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262135    2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:49:09.266478 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262137    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:49:09.266478 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262142    2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:49:09.266478 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262144    2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:49:09.266478 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262147    2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:49:09.266478 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262150    2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:49:09.266478 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262153    2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:49:09.266478 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262155    2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:49:09.266478 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262158    2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:49:09.266478 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262160    2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:49:09.266478 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262163    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:49:09.266478 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262166    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:49:09.266478 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262168    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:49:09.266478 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262171    2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:49:09.266478 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262173    2573 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:49:09.266478 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262176    2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:49:09.266977 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262179    2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:49:09.266977 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262181    2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:49:09.266977 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262184    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:49:09.266977 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262187    2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:49:09.266977 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262189    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:49:09.266977 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262192    2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:49:09.266977 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262195    2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:49:09.266977 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262197    2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:49:09.266977 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262200    2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:49:09.266977 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262202    2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:49:09.266977 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262205    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:49:09.266977 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262208    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:49:09.266977 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262211    2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:49:09.266977 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262214    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:49:09.266977 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262217    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:49:09.266977 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262219    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:49:09.266977 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262222    2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:49:09.266977 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262225    2573 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:49:09.266977 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262228    2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:49:09.266977 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262230    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:49:09.267483 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262233    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:49:09.267483 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262236    2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:49:09.267483 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262238    2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:49:09.267483 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262241    2573 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:49:09.267483 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262243    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:49:09.267483 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262246    2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:49:09.267483 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262248    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:49:09.267483 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262251    2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:49:09.267483 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262253    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:49:09.267483 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262256    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:49:09.267483 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262258    2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:49:09.267483 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262261    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:49:09.267483 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262264    2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:49:09.267483 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262267    2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:49:09.267483 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262269    2573 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:49:09.267483 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262272    2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:49:09.267483 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262274    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:49:09.267483 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262277    2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:49:09.267483 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262280    2573 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:49:09.267483 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262283    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:49:09.267946 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262286    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:49:09.267946 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262289    2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:49:09.267946 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262291    2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:49:09.267946 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262294    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:49:09.267946 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262297    2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:49:09.267946 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262300    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:49:09.267946 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262303    2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:49:09.267946 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262305    2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:49:09.267946 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262308    2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:49:09.267946 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262310    2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:49:09.267946 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262313    2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:49:09.267946 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262315    2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:49:09.267946 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262318    2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:49:09.267946 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262320    2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:49:09.267946 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262323    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:49:09.267946 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262327    2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:49:09.267946 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262331    2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:49:09.267946 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262334    2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:49:09.267946 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262337    2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:49:09.268377 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262340    2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:49:09.268377 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262343    2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:49:09.268377 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262346    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:49:09.268377 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262349    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:49:09.268377 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262351    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:49:09.268377 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262354    2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:49:09.268377 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.262381    2573 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 21:49:09.268377 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262538    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:49:09.268377 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262544    2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:49:09.268377 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262547    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:49:09.268377 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262550    2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:49:09.268377 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262553    2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:49:09.268377 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262556    2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:49:09.268377 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262558    2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:49:09.268377 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262561    2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:49:09.268377 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262564    2573 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:49:09.268747 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262567    2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:49:09.268747 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262570    2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:49:09.268747 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262572    2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:49:09.268747 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262575    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:49:09.268747 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262578    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:49:09.268747 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262581    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:49:09.268747 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262583    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:49:09.268747 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262586    2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:49:09.268747 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262588    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:49:09.268747 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262591    2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:49:09.268747 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262593    2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:49:09.268747 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262596    2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:49:09.268747 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262599    2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:49:09.268747 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262601    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:49:09.268747 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262604    2573 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:49:09.268747 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262607    2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:49:09.268747 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262609    2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:49:09.268747 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262612    2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:49:09.268747 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262614    2573 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:49:09.269198 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262617    2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:49:09.269198 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262619    2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:49:09.269198 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262622    2573 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:49:09.269198 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262625    2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:49:09.269198 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262628    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:49:09.269198 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262630    2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:49:09.269198 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262633    2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:49:09.269198 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262635    2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:49:09.269198 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262638    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:49:09.269198 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262640    2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:49:09.269198 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262643    2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:49:09.269198 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262646    2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:49:09.269198 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262649    2573 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:49:09.269198 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262651    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:49:09.269198 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262654    2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:49:09.269198 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262657    2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:49:09.269198 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262659    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:49:09.269198 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262661    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:49:09.269198 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262664    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:49:09.269198 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262666    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:49:09.269675 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262670    2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:49:09.269675 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262674    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:49:09.269675 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262677    2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:49:09.269675 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262680    2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:49:09.269675 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262683    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:49:09.269675 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262686    2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:49:09.269675 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262689    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:49:09.269675 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262691    2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:49:09.269675 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262694    2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:49:09.269675 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262696    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:49:09.269675 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262699    2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:49:09.269675 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262701    2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:49:09.269675 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262704    2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:49:09.269675 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262706    2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:49:09.269675 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262709    2573 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:49:09.269675 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262711    2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:49:09.269675 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262714    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:49:09.269675 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262717    2573 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:49:09.269675 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262720    2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:49:09.269675 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262722    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:49:09.270195 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262724    2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:49:09.270195 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262727    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:49:09.270195 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262730    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:49:09.270195 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262733    2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:49:09.270195 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262735    2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:49:09.270195 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262738    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:49:09.270195 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262741    2573 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:49:09.270195 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262743    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:49:09.270195 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262746    2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:49:09.270195 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262748    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:49:09.270195 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262750    2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:49:09.270195 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262753    2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:49:09.270195 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262757    2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:49:09.270195 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262760    2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:49:09.270195 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262763    2573 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:49:09.270195 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262766    2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:49:09.270195 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262769    2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:49:09.270195 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262771    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:49:09.270195 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262781    2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:49:09.270664 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262784    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:49:09.270664 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262787    2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:49:09.270664 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262789    2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:49:09.270664 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262792    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:49:09.270664 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262795    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:49:09.270664 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262797    2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:49:09.270664 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262800    2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:49:09.270664 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262802    2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:49:09.270664 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262805    2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:49:09.270664 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262808    2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:49:09.270664 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262810    2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:49:09.270664 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262813    2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:49:09.270664 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262815    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:49:09.270664 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262818    2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:49:09.270664 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262821    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:49:09.270664 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262824    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:49:09.270664 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262826    2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:49:09.270664 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262832    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:49:09.270664 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262835    2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:49:09.271163 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.262859    2573 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 21:49:09.271163 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262900    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:49:09.271163 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262904    2573 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:49:09.271163 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262907    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:49:09.271163 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262910    2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:49:09.271163 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262912    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:49:09.271163 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262915    2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:49:09.271163 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262917    2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:49:09.271163 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262921    2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:49:09.271163 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262925    2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:49:09.271163 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262928    2573 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:49:09.271163 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262932    2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:49:09.271163 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262934    2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:49:09.271163 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262937    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:49:09.271163 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262940    2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:49:09.271554 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262942    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:49:09.271554 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262944    2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:49:09.271554 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262947    2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:49:09.271554 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262949    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:49:09.271554 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262952    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:49:09.271554 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262954    2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:49:09.271554 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262957    2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:49:09.271554 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262959    2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:49:09.271554 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262962    2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:49:09.271554 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262965    2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:49:09.271554 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262968    2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:49:09.271554 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262970    2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:49:09.271554 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262973    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:49:09.271554 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262975    2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:49:09.271554 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262978    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:49:09.271554 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262981    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:49:09.271554 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262984    2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:49:09.271554 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262986    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:49:09.271554 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262989    2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:49:09.272007 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262991    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:49:09.272007 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262994    2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:49:09.272007 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262997    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:49:09.272007 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.262999    2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:49:09.272007 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.263001    2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:49:09.272007 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.263004    2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:49:09.272007 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.263006    2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:49:09.272007 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.263010    2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:49:09.272007 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.263012    2573 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:49:09.272007 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.263015    2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:49:09.272007 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.263021    2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:49:09.272007 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.263024    2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:49:09.272007 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.263032    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:49:09.272007 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.263034    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:49:09.272007 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.263037    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:49:09.272007 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.263039    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:49:09.272007 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.263042    2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:49:09.272007 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.263044    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:49:09.272007 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.263047    2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:49:09.272470 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.263049    2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:49:09.272470 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.263052    2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:49:09.272470 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.263055    2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:49:09.272470 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.263057    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:49:09.272470 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.263060    2573 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:49:09.272470 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.263063    2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:49:09.272470 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.263065    2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:49:09.272470 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.263068    2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:49:09.272470 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.263070    2573 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:49:09.272470 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.263073    2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:49:09.272470 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.263075    2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:49:09.272470 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.263078    2573 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:49:09.272470 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.263081    2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:49:09.272470 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.263084    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:49:09.272470 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.263086    2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:49:09.272470 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.263089    2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:49:09.272470 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.263091    2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:49:09.272470 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.263094    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:49:09.272470 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.263096    2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:49:09.272470 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.263099    2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:49:09.272934 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.263101    2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:49:09.272934 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.263104    2573 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:49:09.272934 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.263106    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:49:09.272934 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.263110    2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:49:09.272934 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.263113    2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:49:09.272934 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.263116    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:49:09.272934 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.263118    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:49:09.272934 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.263121    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:49:09.272934 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.263124    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:49:09.272934 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.263128    2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:49:09.272934 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.263132    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:49:09.272934 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.263136    2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:49:09.272934 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.263138    2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:49:09.272934 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.263141    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:49:09.272934 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.263144    2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:49:09.272934 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.263146    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:49:09.272934 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.263149    2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:49:09.272934 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.263151    2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:49:09.272934 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.263154    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:49:09.272934 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.263156    2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:49:09.273600 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.263159    2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:49:09.273600 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.263162    2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:49:09.273600 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.263164    2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:49:09.273600 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.263167    2573 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:49:09.273600 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.263170    2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:49:09.273600 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.263173    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:49:09.273600 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.263175    2573 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:49:09.273600 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.263178    2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:49:09.273600 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.263181    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:49:09.273600 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.263183    2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:49:09.273600 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.263186    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:49:09.273600 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.263189    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:49:09.273600 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.263192    2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:49:09.273600 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:09.263199    2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:49:09.273600 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.263294    2573 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 21:49:09.273600 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.265499    2573 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 30 21:49:09.273600 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.267235    2573 server.go:1014] "Starting client certificate rotation"
Oct 30 21:49:09.273600 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.267341    2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 21:49:09.273600 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.268618    2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 21:49:09.274068 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.269809    2573 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 21:49:09.274068 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.271063    2573 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 21:49:09.284066 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.284047    2573 log.go:25] "Validated CRI v1 runtime API"
Oct 30 21:49:09.287830 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.287812    2573 log.go:25] "Validated CRI v1 image API"
Oct 30 21:49:09.288688 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.288663    2573 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 21:49:09.295903 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.295881    2573 fs.go:135] Filesystem UUIDs: map[3ce27a66-195d-48d9-927b-7c384f635460:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 c97bc70d-24ac-4265-9008-ed080b7fcaa1:/dev/nvme0n1p3]
Oct 30 21:49:09.295983 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.295900    2573 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 30 21:49:09.302786 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.302660    2573 manager.go:217] Machine: {Timestamp:2026-10-30 21:49:09.300779403 +0000 UTC m=+0.423404122 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099364 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2af5d5f1ca4eee0c8afb3964f4c493 SystemUUID:ec2af5d5-f1ca-4eee-0c8a-fb3964f4c493 BootID:a4655aac-a25f-4893-8d2e-73f95ef98b31 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:06:ff:cf:e5:4f:8b Speed:0 Mtu:9001} {Name:ens5 MacAddress:06:ff:cf:e5:4f:8b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:8e:c3:ef:e6:a6:76 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 21:49:09.303373 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.303362    2573 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 21:49:09.303468 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.303456    2573 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 21:49:09.307269 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.307239    2573 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 21:49:09.307395 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.307271    2573 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-245.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 21:49:09.307458 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.307411    2573 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 21:49:09.307458 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.307419    2573 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 21:49:09.307509 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.307459    2573 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 21:49:09.307509 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.307478    2573 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 21:49:09.308524 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.308514    2573 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 21:49:09.312062 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.312051    2573 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 21:49:09.312177 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.312168    2573 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 21:49:09.312960 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.312952    2573 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 21:49:09.312997 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.312965    2573 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 21:49:09.312997 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.312974    2573 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 21:49:09.312997 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.312984    2573 kubelet.go:405] "Adding apiserver pod source"
Oct 30 21:49:09.312997 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.312991    2573 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 21:49:09.313290 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.313278    2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 21:49:09.316959 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.316941    2573 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 21:49:09.323106 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.323091    2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 21:49:09.324290 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.324278    2573 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 21:49:09.324341 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.324308    2573 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 21:49:09.324473 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.324463    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 21:49:09.324528 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.324479    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 21:49:09.324528 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.324485    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 21:49:09.324528 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.324491    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 21:49:09.324528 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.324497    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 21:49:09.324528 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.324502    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 21:49:09.325148 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.325140    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 21:49:09.325177 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.325149    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 21:49:09.325177 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.325156    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 21:49:09.325177 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.325163    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 21:49:09.325177 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.325172    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 21:49:09.325291 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.325181    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 21:49:09.326899 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.326888    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 21:49:09.326899 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.326898    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 21:49:09.330563 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.330544    2573 server.go:1263] "Started kubelet"
Oct 30 21:49:09.330646 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.330594    2573 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 21:49:09.331151 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.330642    2573 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 21:49:09.331212 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.331173    2573 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 21:49:09.331333 ip-10-0-128-245 systemd[1]: Started Kubernetes Kubelet.
Oct 30 21:49:09.334431 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.334416    2573 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 21:49:09.335135 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.335116    2573 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 21:49:09.338944 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:09.338923    2573 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 30 21:49:09.340710 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.340691    2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 21:49:09.341294 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.341281    2573 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 21:49:09.341599 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:09.341447    2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-245.us-west-2.compute.internal\" not found"
Oct 30 21:49:09.342006 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.341988    2573 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 21:49:09.342006 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.341990    2573 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 21:49:09.342151 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.342016    2573 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 21:49:09.342151 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.342106    2573 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 21:49:09.342151 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.342113    2573 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 21:49:09.342369 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.342353    2573 factory.go:55] Registering systemd factory
Oct 30 21:49:09.342430 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.342371    2573 factory.go:223] Registration of the systemd container factory successfully
Oct 30 21:49:09.342635 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.342622    2573 factory.go:153] Registering CRI-O factory
Oct 30 21:49:09.342635 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.342636    2573 factory.go:223] Registration of the crio container factory successfully
Oct 30 21:49:09.342750 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.342689    2573 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 21:49:09.342750 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.342708    2573 factory.go:103] Registering Raw factory
Oct 30 21:49:09.342750 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.342723    2573 manager.go:1199] Started watching for new ooms in manager
Oct 30 21:49:09.345653 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.344287    2573 manager.go:319] Starting recovery of all containers
Oct 30 21:49:09.355046 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.355024    2573 manager.go:324] Recovery completed
Oct 30 21:49:09.358682 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.358668    2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:49:09.360202 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.360180    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-245.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:49:09.360247 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.360219    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-245.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:49:09.360247 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.360231    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-245.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:49:09.360740 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.360725    2573 cpu_manager.go:226] "Starting" policy="none"
Oct 30 21:49:09.360774 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.360740    2573 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 21:49:09.360774 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.360755    2573 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 21:49:09.361978 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.360786    2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:49:09.361978 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.361472    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-245.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:49:09.361978 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.361502    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-245.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:49:09.361978 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.361513    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-245.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:49:09.362292 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.362281    2573 policy_none.go:50] "Start"
Oct 30 21:49:09.362323 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.362296    2573 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 21:49:09.362323 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.362307    2573 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 21:49:09.363623 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.363613    2573 policy_none.go:44] "Start"
Oct 30 21:49:09.401850 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.401694    2573 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 21:49:09.414695 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.414680    2573 manager.go:341] "Starting Device Plugin manager"
Oct 30 21:49:09.414797 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:09.414727    2573 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 21:49:09.414797 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.414739    2573 server.go:87] "Starting device plugin registration server"
Oct 30 21:49:09.414957 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.414947    2573 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 21:49:09.414998 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.414976    2573 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 21:49:09.415095 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.415073    2573 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 21:49:09.415206 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.415162    2573 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 21:49:09.415206 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.415170    2573 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 21:49:09.415662 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:09.415640    2573 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 21:49:09.415727 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:09.415705    2573 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-245.us-west-2.compute.internal\" not found"
Oct 30 21:49:09.466555 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.466535    2573 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-245.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 30 21:49:09.467825 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:09.466547    2573 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-128-245.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 30 21:49:09.478388 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.478365    2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 21:49:09.479558 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.479531    2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 21:49:09.479558 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.479559    2573 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 21:49:09.479691 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.479578    2573 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 21:49:09.479691 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:09.479616    2573 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Oct 30 21:49:09.513267 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:09.512015    2573 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-245.us-west-2.compute.internal.18e36d060a1d5f46  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-245.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-245.us-west-2.compute.internal,},FirstTimestamp:2026-10-30 21:49:09.33051783 +0000 UTC m=+0.453142548,LastTimestamp:2026-10-30 21:49:09.33051783 +0000 UTC m=+0.453142548,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-245.us-west-2.compute.internal,}"
Oct 30 21:49:09.515077 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.515057    2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:49:09.515836 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.515823    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-245.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:49:09.515910 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.515850    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-245.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:49:09.515910 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.515860    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-245.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:49:09.515910 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.515882    2573 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-128-245.us-west-2.compute.internal"
Oct 30 21:49:09.533143 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.533123    2573 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 21:49:09.533226 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.533161    2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-nbpp2"
Oct 30 21:49:09.535743 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:09.535699    2573 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Oct 30 21:49:09.535743 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:09.535698    2573 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-245.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Oct 30 21:49:09.579962 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.579943    2573 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-128-245.us-west-2.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-245.us-west-2.compute.internal"]
Oct 30 21:49:09.580030 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.579991    2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:49:09.580719 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.580699    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-245.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:49:09.580776 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.580734    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-245.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:49:09.580776 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.580744    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-245.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:49:09.582702 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.582690    2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:49:09.582844 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.582830    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-245.us-west-2.compute.internal"
Oct 30 21:49:09.582901 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.582859    2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:49:09.583352 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.583339    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-245.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:49:09.583467 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.583364    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-245.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:49:09.583467 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.583375    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-245.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:49:09.583467 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.583343    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-245.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:49:09.583467 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.583438    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-245.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:49:09.583467 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.583448    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-245.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:49:09.584899 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.584868    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-245.us-west-2.compute.internal"
Oct 30 21:49:09.584899 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.584898    2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:49:09.585770 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.585754    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-245.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:49:09.585864 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.585796    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-245.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:49:09.585864 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.585815    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-245.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:49:09.606260 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:09.606242    2573 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-245.us-west-2.compute.internal\" not found" node="ip-10-0-128-245.us-west-2.compute.internal"
Oct 30 21:49:09.611755 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:09.611739    2573 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-245.us-west-2.compute.internal\" not found" node="ip-10-0-128-245.us-west-2.compute.internal"
Oct 30 21:49:09.618998 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.618978    2573 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:49:09.619088 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.619033    2573 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:49:09.624643 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.624630    2573 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-128-245.us-west-2.compute.internal"
Oct 30 21:49:09.624722 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:09.624649    2573 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-245.us-west-2.compute.internal\": node \"ip-10-0-128-245.us-west-2.compute.internal\" not found"
Oct 30 21:49:09.644735 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.644710    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bb28c8c054a6af006e24e10c141990f3-config\") pod \"kube-apiserver-proxy-ip-10-0-128-245.us-west-2.compute.internal\" (UID: \"bb28c8c054a6af006e24e10c141990f3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-245.us-west-2.compute.internal"
Oct 30 21:49:09.644818 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.644740    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d8412ee5d9c7152e66cb6933cbd6fbc8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-245.us-west-2.compute.internal\" (UID: \"d8412ee5d9c7152e66cb6933cbd6fbc8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-245.us-west-2.compute.internal"
Oct 30 21:49:09.644818 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.644766    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d8412ee5d9c7152e66cb6933cbd6fbc8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-245.us-west-2.compute.internal\" (UID: \"d8412ee5d9c7152e66cb6933cbd6fbc8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-245.us-west-2.compute.internal"
Oct 30 21:49:09.744742 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:09.744717    2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-245.us-west-2.compute.internal\" not found"
Oct 30 21:49:09.745760 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.745745    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d8412ee5d9c7152e66cb6933cbd6fbc8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-245.us-west-2.compute.internal\" (UID: \"d8412ee5d9c7152e66cb6933cbd6fbc8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-245.us-west-2.compute.internal"
Oct 30 21:49:09.745812 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.745768    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d8412ee5d9c7152e66cb6933cbd6fbc8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-245.us-west-2.compute.internal\" (UID: \"d8412ee5d9c7152e66cb6933cbd6fbc8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-245.us-west-2.compute.internal"
Oct 30 21:49:09.745812 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.745787    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bb28c8c054a6af006e24e10c141990f3-config\") pod \"kube-apiserver-proxy-ip-10-0-128-245.us-west-2.compute.internal\" (UID: \"bb28c8c054a6af006e24e10c141990f3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-245.us-west-2.compute.internal"
Oct 30 21:49:09.745882 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.745841    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bb28c8c054a6af006e24e10c141990f3-config\") pod \"kube-apiserver-proxy-ip-10-0-128-245.us-west-2.compute.internal\" (UID: \"bb28c8c054a6af006e24e10c141990f3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-245.us-west-2.compute.internal"
Oct 30 21:49:09.745882 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.745850    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d8412ee5d9c7152e66cb6933cbd6fbc8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-245.us-west-2.compute.internal\" (UID: \"d8412ee5d9c7152e66cb6933cbd6fbc8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-245.us-west-2.compute.internal"
Oct 30 21:49:09.745943 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.745874    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d8412ee5d9c7152e66cb6933cbd6fbc8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-245.us-west-2.compute.internal\" (UID: \"d8412ee5d9c7152e66cb6933cbd6fbc8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-245.us-west-2.compute.internal"
Oct 30 21:49:09.845765 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:09.845701    2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-245.us-west-2.compute.internal\" not found"
Oct 30 21:49:09.909938 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.909896    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-245.us-west-2.compute.internal"
Oct 30 21:49:09.921054 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:09.921035    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-245.us-west-2.compute.internal"
Oct 30 21:49:09.946670 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:09.946646    2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-245.us-west-2.compute.internal\" not found"
Oct 30 21:49:10.047420 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:10.047387    2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-245.us-west-2.compute.internal\" not found"
Oct 30 21:49:10.148020 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:10.147951    2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-245.us-west-2.compute.internal\" not found"
Oct 30 21:49:10.248458 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:10.248429    2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-245.us-west-2.compute.internal\" not found"
Oct 30 21:49:10.265919 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:10.265899    2573 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 30 21:49:10.266089 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:10.266007    2573 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a8a1717a8546b4318bd8725a26f164b5-359068c9b1489aca.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-128-245.us-west-2.compute.internal.18e36d060be25aa8\": read tcp 10.0.128.245:52408->52.15.155.115:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-128-245.us-west-2.compute.internal.18e36d060be25aa8  default   5394 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-245.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-128-245.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-128-245.us-west-2.compute.internal,},FirstTimestamp:2026-10-30 21:49:09 +0000 UTC,LastTimestamp:2026-10-30 21:49:09.580719592 +0000 UTC m=+0.703344311,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-245.us-west-2.compute.internal,}"
Oct 30 21:49:10.266147 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:10.266071    2573 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 21:49:10.266147 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:10.266074    2573 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 21:49:10.349298 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:10.349265    2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-245.us-west-2.compute.internal\" not found"
Oct 30 21:49:10.450394 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:10.450319    2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-245.us-west-2.compute.internal\" not found"
Oct 30 21:49:10.538704 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:10.538659    2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 21:44:10 +0000 UTC" deadline="2028-07-20 06:56:25.423496872 +0000 UTC"
Oct 30 21:49:10.538704 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:10.538700    2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15081h7m14.884799882s"
Oct 30 21:49:10.550814 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:10.550787    2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-245.us-west-2.compute.internal\" not found"
Oct 30 21:49:10.651982 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:10.651956    2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-245.us-west-2.compute.internal\" not found"
Oct 30 21:49:10.683223 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:10.683197    2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:49:10.684194 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:10.684179    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-245.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:49:10.684269 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:10.684209    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-245.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:49:10.684269 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:10.684220    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-245.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:49:10.737301 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:10.737241    2573 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:49:10.738218 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:10.738203    2573 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:49:10.742638 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:10.742620    2573 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-245.us-west-2.compute.internal"
Oct 30 21:49:10.806605 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:10.806585    2573 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 21:49:10.807568 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:10.807553    2573 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-245.us-west-2.compute.internal"
Oct 30 21:49:10.867865 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:10.867835    2573 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 21:49:11.125495 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:11.125415    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb28c8c054a6af006e24e10c141990f3.slice/crio-3bd252982df6be8fd26cdcd2a455bb58da4c3b32f8bbbdc0df4c8a46ca2826fb WatchSource:0}: Error finding container 3bd252982df6be8fd26cdcd2a455bb58da4c3b32f8bbbdc0df4c8a46ca2826fb: Status 404 returned error can't find the container with id 3bd252982df6be8fd26cdcd2a455bb58da4c3b32f8bbbdc0df4c8a46ca2826fb
Oct 30 21:49:11.125859 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:11.125831    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8412ee5d9c7152e66cb6933cbd6fbc8.slice/crio-ffcebcbe6a69d7a5b1acce836ecf6509f9155e88b6bd05c7b2f2dc9c025ae664 WatchSource:0}: Error finding container ffcebcbe6a69d7a5b1acce836ecf6509f9155e88b6bd05c7b2f2dc9c025ae664: Status 404 returned error can't find the container with id ffcebcbe6a69d7a5b1acce836ecf6509f9155e88b6bd05c7b2f2dc9c025ae664
Oct 30 21:49:11.130941 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.130922    2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 21:49:11.316758 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.316677    2573 apiserver.go:52] "Watching apiserver"
Oct 30 21:49:11.343159 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.343114    2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 30 21:49:11.423010 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.422982    2573 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 21:49:11.423296 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.423274    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-gmbbw","kube-system/global-pull-secret-syncer-cck7d","kube-system/konnectivity-agent-w72c9","openshift-dns/node-resolver-gbpvx","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lrc2c","openshift-cluster-node-tuning-operator/tuned-22pl2","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-245.us-west-2.compute.internal","openshift-multus/multus-additional-cni-plugins-vrtnk","openshift-multus/multus-p7zpx","openshift-multus/network-metrics-daemon-xbh9m","openshift-network-diagnostics/network-check-target-w9tx8","openshift-network-operator/iptables-alerter-t24lz","openshift-ovn-kubernetes/ovnkube-node-ml9xf","kube-system/kube-apiserver-proxy-ip-10-0-128-245.us-west-2.compute.internal"]
Oct 30 21:49:11.425970 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.425953    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xbh9m"
Oct 30 21:49:11.426060 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:11.426042    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xbh9m" podUID="a6980f40-89cf-4f3d-8cd5-91969ae5a276"
Oct 30 21:49:11.427496 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.427481    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf"
Oct 30 21:49:11.428945 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.428922    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-w72c9"
Oct 30 21:49:11.432513 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.430633    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gbpvx"
Oct 30 21:49:11.432832 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.432810    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lrc2c"
Oct 30 21:49:11.434405 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.434386    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-22pl2"
Oct 30 21:49:11.436055 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.436037    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-p7zpx"
Oct 30 21:49:11.437707 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.437687    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-vrtnk"
Oct 30 21:49:11.439803 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.439780    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cck7d"
Oct 30 21:49:11.439910 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:11.439874    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cck7d" podUID="b954bab2-9dd7-4b39-bda6-3fec87381aff"
Oct 30 21:49:11.441429 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.441393    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-gmbbw"
Oct 30 21:49:11.443007 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.442988    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w9tx8"
Oct 30 21:49:11.443101 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:11.443046    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w9tx8" podUID="55619a2e-f6e9-4719-b362-30b69d553967"
Oct 30 21:49:11.444574 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.444556    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-t24lz"
Oct 30 21:49:11.460664 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.460645    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnt2l\" (UniqueName: \"kubernetes.io/projected/924394f7-5b3b-4767-a1ff-f1d310ece4ed-kube-api-access-qnt2l\") pod \"aws-ebs-csi-driver-node-lrc2c\" (UID: \"924394f7-5b3b-4767-a1ff-f1d310ece4ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lrc2c"
Oct 30 21:49:11.460748 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.460671    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/41fe6d63-9316-4916-b042-87ba7e602167-host-run-netns\") pod \"ovnkube-node-ml9xf\" (UID: \"41fe6d63-9316-4916-b042-87ba7e602167\") " pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf"
Oct 30 21:49:11.460748 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.460689    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/41fe6d63-9316-4916-b042-87ba7e602167-run-openvswitch\") pod \"ovnkube-node-ml9xf\" (UID: \"41fe6d63-9316-4916-b042-87ba7e602167\") " pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf"
Oct 30 21:49:11.460748 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.460705    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/41fe6d63-9316-4916-b042-87ba7e602167-ovnkube-config\") pod \"ovnkube-node-ml9xf\" (UID: \"41fe6d63-9316-4916-b042-87ba7e602167\") " pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf"
Oct 30 21:49:11.460884 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.460769    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/41fe6d63-9316-4916-b042-87ba7e602167-ovn-node-metrics-cert\") pod \"ovnkube-node-ml9xf\" (UID: \"41fe6d63-9316-4916-b042-87ba7e602167\") " pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf"
Oct 30 21:49:11.460884 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.460805    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5f468f3c-bf3d-437c-8ec8-a5d56f448d99-host-run-k8s-cni-cncf-io\") pod \"multus-p7zpx\" (UID: \"5f468f3c-bf3d-437c-8ec8-a5d56f448d99\") " pod="openshift-multus/multus-p7zpx"
Oct 30 21:49:11.460884 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.460822    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5f468f3c-bf3d-437c-8ec8-a5d56f448d99-hostroot\") pod \"multus-p7zpx\" (UID: \"5f468f3c-bf3d-437c-8ec8-a5d56f448d99\") " pod="openshift-multus/multus-p7zpx"
Oct 30 21:49:11.460884 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.460844    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b97qt\" (UniqueName: \"kubernetes.io/projected/82f75dd5-40cd-4cb3-9111-6213d6b43679-kube-api-access-b97qt\") pod \"node-ca-gmbbw\" (UID: \"82f75dd5-40cd-4cb3-9111-6213d6b43679\") " pod="openshift-image-registry/node-ca-gmbbw"
Oct 30 21:49:11.460884 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.460873    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5f468f3c-bf3d-437c-8ec8-a5d56f448d99-host-run-netns\") pod \"multus-p7zpx\" (UID: \"5f468f3c-bf3d-437c-8ec8-a5d56f448d99\") " pod="openshift-multus/multus-p7zpx"
Oct 30 21:49:11.461037 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.460899    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5f468f3c-bf3d-437c-8ec8-a5d56f448d99-host-var-lib-kubelet\") pod \"multus-p7zpx\" (UID: \"5f468f3c-bf3d-437c-8ec8-a5d56f448d99\") " pod="openshift-multus/multus-p7zpx"
Oct 30 21:49:11.461037 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.460916    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5f468f3c-bf3d-437c-8ec8-a5d56f448d99-host-run-multus-certs\") pod \"multus-p7zpx\" (UID: \"5f468f3c-bf3d-437c-8ec8-a5d56f448d99\") " pod="openshift-multus/multus-p7zpx"
Oct 30 21:49:11.461037 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.460932    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5f468f3c-bf3d-437c-8ec8-a5d56f448d99-etc-kubernetes\") pod \"multus-p7zpx\" (UID: \"5f468f3c-bf3d-437c-8ec8-a5d56f448d99\") " pod="openshift-multus/multus-p7zpx"
Oct 30 21:49:11.461037 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.460948    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7a6305de-8fd1-4d3e-9628-8c4cda07f379-system-cni-dir\") pod \"multus-additional-cni-plugins-vrtnk\" (UID: \"7a6305de-8fd1-4d3e-9628-8c4cda07f379\") " pod="openshift-multus/multus-additional-cni-plugins-vrtnk"
Oct 30 21:49:11.461037 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.460984    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7a6305de-8fd1-4d3e-9628-8c4cda07f379-cni-binary-copy\") pod \"multus-additional-cni-plugins-vrtnk\" (UID: \"7a6305de-8fd1-4d3e-9628-8c4cda07f379\") " pod="openshift-multus/multus-additional-cni-plugins-vrtnk"
Oct 30 21:49:11.461037 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.461020    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d947d3db-ab6e-4874-8817-f8c74be138a1-etc-modprobe-d\") pod \"tuned-22pl2\" (UID: \"d947d3db-ab6e-4874-8817-f8c74be138a1\") " pod="openshift-cluster-node-tuning-operator/tuned-22pl2"
Oct 30 21:49:11.461209 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.461046    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d947d3db-ab6e-4874-8817-f8c74be138a1-host\") pod \"tuned-22pl2\" (UID: \"d947d3db-ab6e-4874-8817-f8c74be138a1\") " pod="openshift-cluster-node-tuning-operator/tuned-22pl2"
Oct 30 21:49:11.461209 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.461074    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czfxj\" (UniqueName: \"kubernetes.io/projected/a6980f40-89cf-4f3d-8cd5-91969ae5a276-kube-api-access-czfxj\") pod \"network-metrics-daemon-xbh9m\" (UID: \"a6980f40-89cf-4f3d-8cd5-91969ae5a276\") " pod="openshift-multus/network-metrics-daemon-xbh9m"
Oct 30 21:49:11.461209 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.461094    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/41fe6d63-9316-4916-b042-87ba7e602167-host-cni-bin\") pod \"ovnkube-node-ml9xf\" (UID: \"41fe6d63-9316-4916-b042-87ba7e602167\") " pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf"
Oct 30 21:49:11.461209 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.461109    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/41fe6d63-9316-4916-b042-87ba7e602167-ovnkube-script-lib\") pod \"ovnkube-node-ml9xf\" (UID: \"41fe6d63-9316-4916-b042-87ba7e602167\") " pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf"
Oct 30 21:49:11.461209 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.461138    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0ad59239-edcc-46a4-a5a1-1debb63e3a5d-konnectivity-ca\") pod \"konnectivity-agent-w72c9\" (UID: \"0ad59239-edcc-46a4-a5a1-1debb63e3a5d\") " pod="kube-system/konnectivity-agent-w72c9"
Oct 30 21:49:11.461209 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.461159    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/924394f7-5b3b-4767-a1ff-f1d310ece4ed-sys-fs\") pod \"aws-ebs-csi-driver-node-lrc2c\" (UID: \"924394f7-5b3b-4767-a1ff-f1d310ece4ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lrc2c"
Oct 30 21:49:11.461209 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.461183    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d947d3db-ab6e-4874-8817-f8c74be138a1-etc-systemd\") pod \"tuned-22pl2\" (UID: \"d947d3db-ab6e-4874-8817-f8c74be138a1\") " pod="openshift-cluster-node-tuning-operator/tuned-22pl2"
Oct 30 21:49:11.461472 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.461208    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d947d3db-ab6e-4874-8817-f8c74be138a1-tmp\") pod \"tuned-22pl2\" (UID: \"d947d3db-ab6e-4874-8817-f8c74be138a1\") " pod="openshift-cluster-node-tuning-operator/tuned-22pl2"
Oct 30 21:49:11.461472 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.461239    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxd2f\" (UniqueName: \"kubernetes.io/projected/32d63b18-0fcb-4e4a-a24c-ee044dcb5a90-kube-api-access-pxd2f\") pod \"node-resolver-gbpvx\" (UID: \"32d63b18-0fcb-4e4a-a24c-ee044dcb5a90\") " pod="openshift-dns/node-resolver-gbpvx"
Oct 30 21:49:11.461472 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.461269    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/924394f7-5b3b-4767-a1ff-f1d310ece4ed-registration-dir\") pod \"aws-ebs-csi-driver-node-lrc2c\" (UID: \"924394f7-5b3b-4767-a1ff-f1d310ece4ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lrc2c"
Oct 30 21:49:11.461472 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.461300    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5f468f3c-bf3d-437c-8ec8-a5d56f448d99-system-cni-dir\") pod \"multus-p7zpx\" (UID: \"5f468f3c-bf3d-437c-8ec8-a5d56f448d99\") " pod="openshift-multus/multus-p7zpx"
Oct 30 21:49:11.461472 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.461323    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32d63b18-0fcb-4e4a-a24c-ee044dcb5a90-tmp-dir\") pod \"node-resolver-gbpvx\" (UID: \"32d63b18-0fcb-4e4a-a24c-ee044dcb5a90\") " pod="openshift-dns/node-resolver-gbpvx"
Oct 30 21:49:11.461472 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.461338    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b954bab2-9dd7-4b39-bda6-3fec87381aff-kubelet-config\") pod \"global-pull-secret-syncer-cck7d\" (UID: \"b954bab2-9dd7-4b39-bda6-3fec87381aff\") " pod="kube-system/global-pull-secret-syncer-cck7d"
Oct 30 21:49:11.461472 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.461401    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b954bab2-9dd7-4b39-bda6-3fec87381aff-dbus\") pod \"global-pull-secret-syncer-cck7d\" (UID: \"b954bab2-9dd7-4b39-bda6-3fec87381aff\") " pod="kube-system/global-pull-secret-syncer-cck7d"
Oct 30 21:49:11.461472 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.461438    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7a6305de-8fd1-4d3e-9628-8c4cda07f379-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vrtnk\" (UID: \"7a6305de-8fd1-4d3e-9628-8c4cda07f379\") " pod="openshift-multus/multus-additional-cni-plugins-vrtnk"
Oct 30 21:49:11.461876 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.461486    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/924394f7-5b3b-4767-a1ff-f1d310ece4ed-device-dir\") pod \"aws-ebs-csi-driver-node-lrc2c\" (UID: \"924394f7-5b3b-4767-a1ff-f1d310ece4ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lrc2c"
Oct 30 21:49:11.461876 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.461530    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d947d3db-ab6e-4874-8817-f8c74be138a1-etc-sysctl-conf\") pod \"tuned-22pl2\" (UID: \"d947d3db-ab6e-4874-8817-f8c74be138a1\") " pod="openshift-cluster-node-tuning-operator/tuned-22pl2"
Oct 30 21:49:11.461876 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.461547    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d947d3db-ab6e-4874-8817-f8c74be138a1-var-lib-kubelet\") pod \"tuned-22pl2\" (UID: \"d947d3db-ab6e-4874-8817-f8c74be138a1\") " pod="openshift-cluster-node-tuning-operator/tuned-22pl2"
Oct 30 21:49:11.461876 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.461566    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/41fe6d63-9316-4916-b042-87ba7e602167-node-log\") pod \"ovnkube-node-ml9xf\" (UID: \"41fe6d63-9316-4916-b042-87ba7e602167\") " pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf"
Oct 30 21:49:11.461876 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.461600    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/41fe6d63-9316-4916-b042-87ba7e602167-host-cni-netd\") pod \"ovnkube-node-ml9xf\" (UID: \"41fe6d63-9316-4916-b042-87ba7e602167\") " pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf"
Oct 30 21:49:11.461876 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.461624    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7a6305de-8fd1-4d3e-9628-8c4cda07f379-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vrtnk\" (UID: \"7a6305de-8fd1-4d3e-9628-8c4cda07f379\") " pod="openshift-multus/multus-additional-cni-plugins-vrtnk"
Oct 30 21:49:11.461876 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.461662    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d947d3db-ab6e-4874-8817-f8c74be138a1-sys\") pod \"tuned-22pl2\" (UID: \"d947d3db-ab6e-4874-8817-f8c74be138a1\") " pod="openshift-cluster-node-tuning-operator/tuned-22pl2"
Oct 30 21:49:11.461876 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.461683    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b954bab2-9dd7-4b39-bda6-3fec87381aff-original-pull-secret\") pod \"global-pull-secret-syncer-cck7d\" (UID: \"b954bab2-9dd7-4b39-bda6-3fec87381aff\") " pod="kube-system/global-pull-secret-syncer-cck7d"
Oct 30 21:49:11.461876 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.461715    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/41fe6d63-9316-4916-b042-87ba7e602167-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ml9xf\" (UID: \"41fe6d63-9316-4916-b042-87ba7e602167\") " pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf"
Oct 30 21:49:11.461876 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.461754    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/41fe6d63-9316-4916-b042-87ba7e602167-env-overrides\") pod \"ovnkube-node-ml9xf\" (UID: \"41fe6d63-9316-4916-b042-87ba7e602167\") " pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf"
Oct 30 21:49:11.461876 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.461778    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5f468f3c-bf3d-437c-8ec8-a5d56f448d99-cnibin\") pod \"multus-p7zpx\" (UID: \"5f468f3c-bf3d-437c-8ec8-a5d56f448d99\") " pod="openshift-multus/multus-p7zpx"
Oct 30 21:49:11.461876 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.461805    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5f468f3c-bf3d-437c-8ec8-a5d56f448d99-cni-binary-copy\") pod \"multus-p7zpx\" (UID: \"5f468f3c-bf3d-437c-8ec8-a5d56f448d99\") " pod="openshift-multus/multus-p7zpx"
Oct 30 21:49:11.461876 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.461834    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5f468f3c-bf3d-437c-8ec8-a5d56f448d99-multus-daemon-config\") pod \"multus-p7zpx\" (UID: \"5f468f3c-bf3d-437c-8ec8-a5d56f448d99\") " pod="openshift-multus/multus-p7zpx"
Oct 30 21:49:11.462298 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.461883    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/41fe6d63-9316-4916-b042-87ba7e602167-systemd-units\") pod \"ovnkube-node-ml9xf\" (UID: \"41fe6d63-9316-4916-b042-87ba7e602167\") " pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf"
Oct 30 21:49:11.462298 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.461917    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6x28\" (UniqueName: \"kubernetes.io/projected/55619a2e-f6e9-4719-b362-30b69d553967-kube-api-access-j6x28\") pod \"network-check-target-w9tx8\" (UID: \"55619a2e-f6e9-4719-b362-30b69d553967\") " pod="openshift-network-diagnostics/network-check-target-w9tx8"
Oct 30 21:49:11.462298 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.461946    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/924394f7-5b3b-4767-a1ff-f1d310ece4ed-etc-selinux\") pod \"aws-ebs-csi-driver-node-lrc2c\" (UID: \"924394f7-5b3b-4767-a1ff-f1d310ece4ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lrc2c"
Oct 30 21:49:11.462298 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.461973    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d947d3db-ab6e-4874-8817-f8c74be138a1-etc-sysconfig\") pod \"tuned-22pl2\" (UID: \"d947d3db-ab6e-4874-8817-f8c74be138a1\") " pod="openshift-cluster-node-tuning-operator/tuned-22pl2"
Oct 30 21:49:11.462298 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.461990    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d947d3db-ab6e-4874-8817-f8c74be138a1-run\") pod \"tuned-22pl2\" (UID: \"d947d3db-ab6e-4874-8817-f8c74be138a1\") " pod="openshift-cluster-node-tuning-operator/tuned-22pl2"
Oct 30 21:49:11.462298 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.462007    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5f468f3c-bf3d-437c-8ec8-a5d56f448d99-host-var-lib-cni-multus\") pod \"multus-p7zpx\" (UID: \"5f468f3c-bf3d-437c-8ec8-a5d56f448d99\") " pod="openshift-multus/multus-p7zpx"
Oct 30 21:49:11.462298 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.462024    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgn7x\" (UniqueName: \"kubernetes.io/projected/5f468f3c-bf3d-437c-8ec8-a5d56f448d99-kube-api-access-rgn7x\") pod \"multus-p7zpx\" (UID: \"5f468f3c-bf3d-437c-8ec8-a5d56f448d99\") " pod="openshift-multus/multus-p7zpx"
Oct 30 21:49:11.462298 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.462048    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7a6305de-8fd1-4d3e-9628-8c4cda07f379-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vrtnk\" (UID: \"7a6305de-8fd1-4d3e-9628-8c4cda07f379\") " pod="openshift-multus/multus-additional-cni-plugins-vrtnk"
Oct 30 21:49:11.462298 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.462074    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d947d3db-ab6e-4874-8817-f8c74be138a1-etc-tuned\") pod \"tuned-22pl2\" (UID: \"d947d3db-ab6e-4874-8817-f8c74be138a1\") " pod="openshift-cluster-node-tuning-operator/tuned-22pl2"
Oct 30 21:49:11.462298 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.462098    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/41fe6d63-9316-4916-b042-87ba7e602167-host-slash\") pod \"ovnkube-node-ml9xf\" (UID: \"41fe6d63-9316-4916-b042-87ba7e602167\") " pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf"
Oct 30 21:49:11.462298 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.462124    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/41fe6d63-9316-4916-b042-87ba7e602167-log-socket\") pod \"ovnkube-node-ml9xf\" (UID: \"41fe6d63-9316-4916-b042-87ba7e602167\") " pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf"
Oct 30 21:49:11.462298 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.462153    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0ad59239-edcc-46a4-a5a1-1debb63e3a5d-agent-certs\") pod \"konnectivity-agent-w72c9\" (UID: \"0ad59239-edcc-46a4-a5a1-1debb63e3a5d\") " pod="kube-system/konnectivity-agent-w72c9"
Oct 30 21:49:11.462298 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.462172    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d947d3db-ab6e-4874-8817-f8c74be138a1-etc-kubernetes\") pod \"tuned-22pl2\" (UID: \"d947d3db-ab6e-4874-8817-f8c74be138a1\") " pod="openshift-cluster-node-tuning-operator/tuned-22pl2"
Oct 30 21:49:11.462298 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.462202    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6980f40-89cf-4f3d-8cd5-91969ae5a276-metrics-certs\") pod \"network-metrics-daemon-xbh9m\" (UID: \"a6980f40-89cf-4f3d-8cd5-91969ae5a276\") " pod="openshift-multus/network-metrics-daemon-xbh9m"
Oct 30 21:49:11.462298 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.462222    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/41fe6d63-9316-4916-b042-87ba7e602167-host-kubelet\") pod \"ovnkube-node-ml9xf\" (UID: \"41fe6d63-9316-4916-b042-87ba7e602167\") " pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf"
Oct 30 21:49:11.462298 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.462237    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5f468f3c-bf3d-437c-8ec8-a5d56f448d99-multus-conf-dir\") pod \"multus-p7zpx\" (UID: \"5f468f3c-bf3d-437c-8ec8-a5d56f448d99\") " pod="openshift-multus/multus-p7zpx"
Oct 30 21:49:11.462962 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.462252    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/924394f7-5b3b-4767-a1ff-f1d310ece4ed-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lrc2c\" (UID: \"924394f7-5b3b-4767-a1ff-f1d310ece4ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lrc2c"
Oct 30 21:49:11.462962 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.462268    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/82f75dd5-40cd-4cb3-9111-6213d6b43679-host\") pod \"node-ca-gmbbw\" (UID: \"82f75dd5-40cd-4cb3-9111-6213d6b43679\") " pod="openshift-image-registry/node-ca-gmbbw"
Oct 30 21:49:11.462962 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.462305    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/41fe6d63-9316-4916-b042-87ba7e602167-run-systemd\") pod \"ovnkube-node-ml9xf\" (UID: \"41fe6d63-9316-4916-b042-87ba7e602167\") " pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf"
Oct 30 21:49:11.462962 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.462325    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/41fe6d63-9316-4916-b042-87ba7e602167-var-lib-openvswitch\") pod \"ovnkube-node-ml9xf\" (UID: \"41fe6d63-9316-4916-b042-87ba7e602167\") " pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf"
Oct 30 21:49:11.462962 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.462340    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5f468f3c-bf3d-437c-8ec8-a5d56f448d99-host-var-lib-cni-bin\") pod \"multus-p7zpx\" (UID: \"5f468f3c-bf3d-437c-8ec8-a5d56f448d99\") " pod="openshift-multus/multus-p7zpx"
Oct 30 21:49:11.462962 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.462376    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d947d3db-ab6e-4874-8817-f8c74be138a1-lib-modules\") pod \"tuned-22pl2\" (UID: \"d947d3db-ab6e-4874-8817-f8c74be138a1\") " pod="openshift-cluster-node-tuning-operator/tuned-22pl2"
Oct 30 21:49:11.462962 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.462422    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwvcm\" (UniqueName: \"kubernetes.io/projected/7a6305de-8fd1-4d3e-9628-8c4cda07f379-kube-api-access-lwvcm\") pod \"multus-additional-cni-plugins-vrtnk\" (UID: \"7a6305de-8fd1-4d3e-9628-8c4cda07f379\") " pod="openshift-multus/multus-additional-cni-plugins-vrtnk"
Oct 30 21:49:11.462962 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.462485    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/32d63b18-0fcb-4e4a-a24c-ee044dcb5a90-hosts-file\") pod \"node-resolver-gbpvx\" (UID: \"32d63b18-0fcb-4e4a-a24c-ee044dcb5a90\") " pod="openshift-dns/node-resolver-gbpvx"
Oct 30 21:49:11.462962 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.462526    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/924394f7-5b3b-4767-a1ff-f1d310ece4ed-socket-dir\") pod \"aws-ebs-csi-driver-node-lrc2c\" (UID: \"924394f7-5b3b-4767-a1ff-f1d310ece4ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lrc2c"
Oct 30 21:49:11.462962 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.462553    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/82f75dd5-40cd-4cb3-9111-6213d6b43679-serviceca\") pod \"node-ca-gmbbw\" (UID: \"82f75dd5-40cd-4cb3-9111-6213d6b43679\") " pod="openshift-image-registry/node-ca-gmbbw"
Oct 30 21:49:11.462962 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.462569    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/41fe6d63-9316-4916-b042-87ba7e602167-etc-openvswitch\") pod \"ovnkube-node-ml9xf\" (UID: \"41fe6d63-9316-4916-b042-87ba7e602167\") " pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf"
Oct 30 21:49:11.462962 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.462615    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/41fe6d63-9316-4916-b042-87ba7e602167-host-run-ovn-kubernetes\") pod \"ovnkube-node-ml9xf\" (UID: \"41fe6d63-9316-4916-b042-87ba7e602167\") " pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf"
Oct 30 21:49:11.462962 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.462641    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hghq\" (UniqueName: \"kubernetes.io/projected/41fe6d63-9316-4916-b042-87ba7e602167-kube-api-access-2hghq\") pod \"ovnkube-node-ml9xf\" (UID: \"41fe6d63-9316-4916-b042-87ba7e602167\") " pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf"
Oct 30 21:49:11.462962 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.462666    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5f468f3c-bf3d-437c-8ec8-a5d56f448d99-multus-cni-dir\") pod \"multus-p7zpx\" (UID: \"5f468f3c-bf3d-437c-8ec8-a5d56f448d99\") " pod="openshift-multus/multus-p7zpx"
Oct 30 21:49:11.462962 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.462713    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5f468f3c-bf3d-437c-8ec8-a5d56f448d99-multus-socket-dir-parent\") pod \"multus-p7zpx\" (UID: \"5f468f3c-bf3d-437c-8ec8-a5d56f448d99\") " pod="openshift-multus/multus-p7zpx"
Oct 30 21:49:11.462962 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.462759    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d947d3db-ab6e-4874-8817-f8c74be138a1-etc-sysctl-d\") pod \"tuned-22pl2\" (UID: \"d947d3db-ab6e-4874-8817-f8c74be138a1\") " pod="openshift-cluster-node-tuning-operator/tuned-22pl2"
Oct 30 21:49:11.463672 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.462783    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkkmr\" (UniqueName: \"kubernetes.io/projected/d947d3db-ab6e-4874-8817-f8c74be138a1-kube-api-access-mkkmr\") pod \"tuned-22pl2\" (UID: \"d947d3db-ab6e-4874-8817-f8c74be138a1\") " pod="openshift-cluster-node-tuning-operator/tuned-22pl2"
Oct 30 21:49:11.463672 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.462828    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/41fe6d63-9316-4916-b042-87ba7e602167-run-ovn\") pod \"ovnkube-node-ml9xf\" (UID: \"41fe6d63-9316-4916-b042-87ba7e602167\") " pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf"
Oct 30 21:49:11.463672 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.462859    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7a6305de-8fd1-4d3e-9628-8c4cda07f379-cnibin\") pod \"multus-additional-cni-plugins-vrtnk\" (UID: \"7a6305de-8fd1-4d3e-9628-8c4cda07f379\") " pod="openshift-multus/multus-additional-cni-plugins-vrtnk"
Oct 30 21:49:11.481735 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.481712    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 21:49:11.481844 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.481777    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 21:49:11.481844 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.481791    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 21:49:11.482030 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.482012    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 21:49:11.482540 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.482361    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 21:49:11.482540 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.482491    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-hm4gw\""
Oct 30 21:49:11.482540 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.482533    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 21:49:11.485132 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.485087    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-245.us-west-2.compute.internal" event={"ID":"d8412ee5d9c7152e66cb6933cbd6fbc8","Type":"ContainerStarted","Data":"ffcebcbe6a69d7a5b1acce836ecf6509f9155e88b6bd05c7b2f2dc9c025ae664"}
Oct 30 21:49:11.486174 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.486155    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-245.us-west-2.compute.internal" event={"ID":"bb28c8c054a6af006e24e10c141990f3","Type":"ContainerStarted","Data":"3bd252982df6be8fd26cdcd2a455bb58da4c3b32f8bbbdc0df4c8a46ca2826fb"}
Oct 30 21:49:11.529702 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.529670    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 21:49:11.532878 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.530740    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 21:49:11.532878 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.531122    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 21:49:11.532878 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.531681    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-6gwn9\""
Oct 30 21:49:11.532878 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.531784    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 21:49:11.532878 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.531919    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 21:49:11.532878 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.532005    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 21:49:11.532878 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.532125    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 21:49:11.532878 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.532311    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-7d2ft\""
Oct 30 21:49:11.532878 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.532408    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 21:49:11.532878 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.532579    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 21:49:11.532878 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.532611    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 21:49:11.532878 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.532690    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 21:49:11.532878 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.532714    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 21:49:11.532878 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.532767    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 21:49:11.532878 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.532839    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 21:49:11.532878 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.532858    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 21:49:11.533649 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.532913    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 21:49:11.533649 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.532948    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-dvh7z\""
Oct 30 21:49:11.533649 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.532620    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 21:49:11.533649 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.533002    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 21:49:11.533649 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.533019    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 21:49:11.533649 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.533028    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-cmhxq\""
Oct 30 21:49:11.533649 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.533028    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 21:49:11.533649 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.533121    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-92jf8\""
Oct 30 21:49:11.533649 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.533179    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-nwhdt\""
Oct 30 21:49:11.533649 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.533213    2573 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 21:49:11.533649 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.533218    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 21:49:11.533649 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.533246    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-gbc4r\""
Oct 30 21:49:11.533649 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.533249    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-9mttd\""
Oct 30 21:49:11.533649 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.533283    2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-s8qqw"
Oct 30 21:49:11.543719 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.543705    2573 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 21:49:11.563714 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.563689    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7a6305de-8fd1-4d3e-9628-8c4cda07f379-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vrtnk\" (UID: \"7a6305de-8fd1-4d3e-9628-8c4cda07f379\") " pod="openshift-multus/multus-additional-cni-plugins-vrtnk"
Oct 30 21:49:11.563804 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.563785    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d947d3db-ab6e-4874-8817-f8c74be138a1-sys\") pod \"tuned-22pl2\" (UID: \"d947d3db-ab6e-4874-8817-f8c74be138a1\") " pod="openshift-cluster-node-tuning-operator/tuned-22pl2"
Oct 30 21:49:11.563865 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.563816    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b954bab2-9dd7-4b39-bda6-3fec87381aff-original-pull-secret\") pod \"global-pull-secret-syncer-cck7d\" (UID: \"b954bab2-9dd7-4b39-bda6-3fec87381aff\") " pod="kube-system/global-pull-secret-syncer-cck7d"
Oct 30 21:49:11.563865 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.563836    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/41fe6d63-9316-4916-b042-87ba7e602167-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ml9xf\" (UID: \"41fe6d63-9316-4916-b042-87ba7e602167\") " pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf"
Oct 30 21:49:11.563865 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.563854    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/41fe6d63-9316-4916-b042-87ba7e602167-env-overrides\") pod \"ovnkube-node-ml9xf\" (UID: \"41fe6d63-9316-4916-b042-87ba7e602167\") " pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf"
Oct 30 21:49:11.564013 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.563878    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5f468f3c-bf3d-437c-8ec8-a5d56f448d99-cnibin\") pod \"multus-p7zpx\" (UID: \"5f468f3c-bf3d-437c-8ec8-a5d56f448d99\") " pod="openshift-multus/multus-p7zpx"
Oct 30 21:49:11.564013 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.563902    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5f468f3c-bf3d-437c-8ec8-a5d56f448d99-cni-binary-copy\") pod \"multus-p7zpx\" (UID: \"5f468f3c-bf3d-437c-8ec8-a5d56f448d99\") " pod="openshift-multus/multus-p7zpx"
Oct 30 21:49:11.564013 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.563927    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5f468f3c-bf3d-437c-8ec8-a5d56f448d99-multus-daemon-config\") pod \"multus-p7zpx\" (UID: \"5f468f3c-bf3d-437c-8ec8-a5d56f448d99\") " pod="openshift-multus/multus-p7zpx"
Oct 30 21:49:11.564013 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.563928    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/41fe6d63-9316-4916-b042-87ba7e602167-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ml9xf\" (UID: \"41fe6d63-9316-4916-b042-87ba7e602167\") " pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf"
Oct 30 21:49:11.564013 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.563968    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d947d3db-ab6e-4874-8817-f8c74be138a1-sys\") pod \"tuned-22pl2\" (UID: \"d947d3db-ab6e-4874-8817-f8c74be138a1\") " pod="openshift-cluster-node-tuning-operator/tuned-22pl2"
Oct 30 21:49:11.564013 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.563975    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/41fe6d63-9316-4916-b042-87ba7e602167-systemd-units\") pod \"ovnkube-node-ml9xf\" (UID: \"41fe6d63-9316-4916-b042-87ba7e602167\") " pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf"
Oct 30 21:49:11.564013 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.563996    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5f468f3c-bf3d-437c-8ec8-a5d56f448d99-cnibin\") pod \"multus-p7zpx\" (UID: \"5f468f3c-bf3d-437c-8ec8-a5d56f448d99\") " pod="openshift-multus/multus-p7zpx"
Oct 30 21:49:11.564013 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.564009    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6x28\" (UniqueName: \"kubernetes.io/projected/55619a2e-f6e9-4719-b362-30b69d553967-kube-api-access-j6x28\") pod \"network-check-target-w9tx8\" (UID: \"55619a2e-f6e9-4719-b362-30b69d553967\") " pod="openshift-network-diagnostics/network-check-target-w9tx8"
Oct 30 21:49:11.564401 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.564037    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/924394f7-5b3b-4767-a1ff-f1d310ece4ed-etc-selinux\") pod \"aws-ebs-csi-driver-node-lrc2c\" (UID: \"924394f7-5b3b-4767-a1ff-f1d310ece4ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lrc2c"
Oct 30 21:49:11.564401 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.564063    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d947d3db-ab6e-4874-8817-f8c74be138a1-etc-sysconfig\") pod \"tuned-22pl2\" (UID: \"d947d3db-ab6e-4874-8817-f8c74be138a1\") " pod="openshift-cluster-node-tuning-operator/tuned-22pl2"
Oct 30 21:49:11.564401 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:11.564079    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 21:49:11.564401 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.564127    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d947d3db-ab6e-4874-8817-f8c74be138a1-etc-sysconfig\") pod \"tuned-22pl2\" (UID: \"d947d3db-ab6e-4874-8817-f8c74be138a1\") " pod="openshift-cluster-node-tuning-operator/tuned-22pl2"
Oct 30 21:49:11.564401 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:11.564159    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b954bab2-9dd7-4b39-bda6-3fec87381aff-original-pull-secret podName:b954bab2-9dd7-4b39-bda6-3fec87381aff nodeName:}" failed. No retries permitted until 2026-10-30 21:49:12.064130362 +0000 UTC m=+3.186755091 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b954bab2-9dd7-4b39-bda6-3fec87381aff-original-pull-secret") pod "global-pull-secret-syncer-cck7d" (UID: "b954bab2-9dd7-4b39-bda6-3fec87381aff") : object "kube-system"/"original-pull-secret" not registered
Oct 30 21:49:11.564401 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.564186    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/924394f7-5b3b-4767-a1ff-f1d310ece4ed-etc-selinux\") pod \"aws-ebs-csi-driver-node-lrc2c\" (UID: \"924394f7-5b3b-4767-a1ff-f1d310ece4ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lrc2c"
Oct 30 21:49:11.564401 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.564209    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/41fe6d63-9316-4916-b042-87ba7e602167-systemd-units\") pod \"ovnkube-node-ml9xf\" (UID: \"41fe6d63-9316-4916-b042-87ba7e602167\") " pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf"
Oct 30 21:49:11.564401 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.564324    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d947d3db-ab6e-4874-8817-f8c74be138a1-run\") pod \"tuned-22pl2\" (UID: \"d947d3db-ab6e-4874-8817-f8c74be138a1\") " pod="openshift-cluster-node-tuning-operator/tuned-22pl2"
Oct 30 21:49:11.564401 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.564356    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5f468f3c-bf3d-437c-8ec8-a5d56f448d99-host-var-lib-cni-multus\") pod \"multus-p7zpx\" (UID: \"5f468f3c-bf3d-437c-8ec8-a5d56f448d99\") " pod="openshift-multus/multus-p7zpx"
Oct 30 21:49:11.564401 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.564381    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgn7x\" (UniqueName: \"kubernetes.io/projected/5f468f3c-bf3d-437c-8ec8-a5d56f448d99-kube-api-access-rgn7x\") pod \"multus-p7zpx\" (UID: \"5f468f3c-bf3d-437c-8ec8-a5d56f448d99\") " pod="openshift-multus/multus-p7zpx"
Oct 30 21:49:11.564917 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.564425    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7a6305de-8fd1-4d3e-9628-8c4cda07f379-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vrtnk\" (UID: \"7a6305de-8fd1-4d3e-9628-8c4cda07f379\") " pod="openshift-multus/multus-additional-cni-plugins-vrtnk"
Oct 30 21:49:11.564917 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.564452    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d947d3db-ab6e-4874-8817-f8c74be138a1-etc-tuned\") pod \"tuned-22pl2\" (UID: \"d947d3db-ab6e-4874-8817-f8c74be138a1\") " pod="openshift-cluster-node-tuning-operator/tuned-22pl2"
Oct 30 21:49:11.564917 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.564459    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/41fe6d63-9316-4916-b042-87ba7e602167-env-overrides\") pod \"ovnkube-node-ml9xf\" (UID: \"41fe6d63-9316-4916-b042-87ba7e602167\") " pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf"
Oct 30 21:49:11.564917 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.564444    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7a6305de-8fd1-4d3e-9628-8c4cda07f379-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vrtnk\" (UID: \"7a6305de-8fd1-4d3e-9628-8c4cda07f379\") " pod="openshift-multus/multus-additional-cni-plugins-vrtnk"
Oct 30 21:49:11.564917 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.564478    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/41fe6d63-9316-4916-b042-87ba7e602167-host-slash\") pod \"ovnkube-node-ml9xf\" (UID: \"41fe6d63-9316-4916-b042-87ba7e602167\") " pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf"
Oct 30 21:49:11.564917 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.564505    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/41fe6d63-9316-4916-b042-87ba7e602167-log-socket\") pod \"ovnkube-node-ml9xf\" (UID: \"41fe6d63-9316-4916-b042-87ba7e602167\") " pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf"
Oct 30 21:49:11.564917 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.564506    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d947d3db-ab6e-4874-8817-f8c74be138a1-run\") pod \"tuned-22pl2\" (UID: \"d947d3db-ab6e-4874-8817-f8c74be138a1\") " pod="openshift-cluster-node-tuning-operator/tuned-22pl2"
Oct 30 21:49:11.564917 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.564565    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/41fe6d63-9316-4916-b042-87ba7e602167-log-socket\") pod \"ovnkube-node-ml9xf\" (UID: \"41fe6d63-9316-4916-b042-87ba7e602167\") " pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf"
Oct 30 21:49:11.564917 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.564563    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5f468f3c-bf3d-437c-8ec8-a5d56f448d99-host-var-lib-cni-multus\") pod \"multus-p7zpx\" (UID: \"5f468f3c-bf3d-437c-8ec8-a5d56f448d99\") " pod="openshift-multus/multus-p7zpx"
Oct 30 21:49:11.564917 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.564646    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/41fe6d63-9316-4916-b042-87ba7e602167-host-slash\") pod \"ovnkube-node-ml9xf\" (UID: \"41fe6d63-9316-4916-b042-87ba7e602167\") " pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf"
Oct 30 21:49:11.564917 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.564683    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0ad59239-edcc-46a4-a5a1-1debb63e3a5d-agent-certs\") pod \"konnectivity-agent-w72c9\" (UID: \"0ad59239-edcc-46a4-a5a1-1debb63e3a5d\") " pod="kube-system/konnectivity-agent-w72c9"
Oct 30 21:49:11.564917 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.564742    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d947d3db-ab6e-4874-8817-f8c74be138a1-etc-kubernetes\") pod \"tuned-22pl2\" (UID: \"d947d3db-ab6e-4874-8817-f8c74be138a1\") " pod="openshift-cluster-node-tuning-operator/tuned-22pl2"
Oct 30 21:49:11.564917 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.564794    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6980f40-89cf-4f3d-8cd5-91969ae5a276-metrics-certs\") pod \"network-metrics-daemon-xbh9m\" (UID: \"a6980f40-89cf-4f3d-8cd5-91969ae5a276\") " pod="openshift-multus/network-metrics-daemon-xbh9m"
Oct 30 21:49:11.564917 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.564820    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/41fe6d63-9316-4916-b042-87ba7e602167-host-kubelet\") pod \"ovnkube-node-ml9xf\" (UID: \"41fe6d63-9316-4916-b042-87ba7e602167\") " pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf"
Oct 30 21:49:11.564917 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.564828    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d947d3db-ab6e-4874-8817-f8c74be138a1-etc-kubernetes\") pod \"tuned-22pl2\" (UID: \"d947d3db-ab6e-4874-8817-f8c74be138a1\") " pod="openshift-cluster-node-tuning-operator/tuned-22pl2"
Oct 30 21:49:11.564917 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.564851    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5f468f3c-bf3d-437c-8ec8-a5d56f448d99-multus-conf-dir\") pod \"multus-p7zpx\" (UID: \"5f468f3c-bf3d-437c-8ec8-a5d56f448d99\") " pod="openshift-multus/multus-p7zpx"
Oct 30 21:49:11.564917 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.564883    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f22e4514-ac6d-481d-8afe-6e2e4cdc3aa1-host-slash\") pod \"iptables-alerter-t24lz\" (UID: \"f22e4514-ac6d-481d-8afe-6e2e4cdc3aa1\") " pod="openshift-network-operator/iptables-alerter-t24lz"
Oct 30 21:49:11.565685 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.564908    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/41fe6d63-9316-4916-b042-87ba7e602167-host-kubelet\") pod \"ovnkube-node-ml9xf\" (UID: \"41fe6d63-9316-4916-b042-87ba7e602167\") " pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf"
Oct 30 21:49:11.565685 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.564909    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/924394f7-5b3b-4767-a1ff-f1d310ece4ed-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lrc2c\" (UID: \"924394f7-5b3b-4767-a1ff-f1d310ece4ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lrc2c"
Oct 30 21:49:11.565685 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:11.564929    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:49:11.565685 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.564940    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/82f75dd5-40cd-4cb3-9111-6213d6b43679-host\") pod \"node-ca-gmbbw\" (UID: \"82f75dd5-40cd-4cb3-9111-6213d6b43679\") " pod="openshift-image-registry/node-ca-gmbbw"
Oct 30 21:49:11.565685 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.564949    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5f468f3c-bf3d-437c-8ec8-a5d56f448d99-multus-conf-dir\") pod \"multus-p7zpx\" (UID: \"5f468f3c-bf3d-437c-8ec8-a5d56f448d99\") " pod="openshift-multus/multus-p7zpx"
Oct 30 21:49:11.565685 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.564963    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/924394f7-5b3b-4767-a1ff-f1d310ece4ed-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lrc2c\" (UID: \"924394f7-5b3b-4767-a1ff-f1d310ece4ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lrc2c"
Oct 30 21:49:11.565685 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:11.564984    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6980f40-89cf-4f3d-8cd5-91969ae5a276-metrics-certs podName:a6980f40-89cf-4f3d-8cd5-91969ae5a276 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:12.064963945 +0000 UTC m=+3.187588666 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a6980f40-89cf-4f3d-8cd5-91969ae5a276-metrics-certs") pod "network-metrics-daemon-xbh9m" (UID: "a6980f40-89cf-4f3d-8cd5-91969ae5a276") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:49:11.565685 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.565094    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/82f75dd5-40cd-4cb3-9111-6213d6b43679-host\") pod \"node-ca-gmbbw\" (UID: \"82f75dd5-40cd-4cb3-9111-6213d6b43679\") " pod="openshift-image-registry/node-ca-gmbbw"
Oct 30 21:49:11.565685 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.565094    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/41fe6d63-9316-4916-b042-87ba7e602167-run-systemd\") pod \"ovnkube-node-ml9xf\" (UID: \"41fe6d63-9316-4916-b042-87ba7e602167\") " pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf"
Oct 30 21:49:11.565685 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.565141    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/41fe6d63-9316-4916-b042-87ba7e602167-run-systemd\") pod \"ovnkube-node-ml9xf\" (UID: \"41fe6d63-9316-4916-b042-87ba7e602167\") " pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf"
Oct 30 21:49:11.565685 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.565149    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/41fe6d63-9316-4916-b042-87ba7e602167-var-lib-openvswitch\") pod \"ovnkube-node-ml9xf\" (UID: \"41fe6d63-9316-4916-b042-87ba7e602167\") " pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf"
Oct 30 21:49:11.565685 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.565190    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5f468f3c-bf3d-437c-8ec8-a5d56f448d99-host-var-lib-cni-bin\") pod \"multus-p7zpx\" (UID: \"5f468f3c-bf3d-437c-8ec8-a5d56f448d99\") " pod="openshift-multus/multus-p7zpx"
Oct 30 21:49:11.565685 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.565216    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d947d3db-ab6e-4874-8817-f8c74be138a1-lib-modules\") pod \"tuned-22pl2\" (UID: \"d947d3db-ab6e-4874-8817-f8c74be138a1\") " pod="openshift-cluster-node-tuning-operator/tuned-22pl2"
Oct 30 21:49:11.565685 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.565226    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/41fe6d63-9316-4916-b042-87ba7e602167-var-lib-openvswitch\") pod \"ovnkube-node-ml9xf\" (UID: \"41fe6d63-9316-4916-b042-87ba7e602167\") " pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf"
Oct 30 21:49:11.565685 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.565246    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwvcm\" (UniqueName: \"kubernetes.io/projected/7a6305de-8fd1-4d3e-9628-8c4cda07f379-kube-api-access-lwvcm\") pod \"multus-additional-cni-plugins-vrtnk\" (UID: \"7a6305de-8fd1-4d3e-9628-8c4cda07f379\") " pod="openshift-multus/multus-additional-cni-plugins-vrtnk"
Oct 30 21:49:11.565685 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.565228    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5f468f3c-bf3d-437c-8ec8-a5d56f448d99-host-var-lib-cni-bin\") pod \"multus-p7zpx\" (UID: \"5f468f3c-bf3d-437c-8ec8-a5d56f448d99\") " pod="openshift-multus/multus-p7zpx"
Oct 30 21:49:11.565685 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.565272    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/32d63b18-0fcb-4e4a-a24c-ee044dcb5a90-hosts-file\") pod \"node-resolver-gbpvx\" (UID: \"32d63b18-0fcb-4e4a-a24c-ee044dcb5a90\") " pod="openshift-dns/node-resolver-gbpvx"
Oct 30 21:49:11.566538 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.565301    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/924394f7-5b3b-4767-a1ff-f1d310ece4ed-socket-dir\") pod \"aws-ebs-csi-driver-node-lrc2c\" (UID: \"924394f7-5b3b-4767-a1ff-f1d310ece4ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lrc2c"
Oct 30 21:49:11.566538 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.565329    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/82f75dd5-40cd-4cb3-9111-6213d6b43679-serviceca\") pod \"node-ca-gmbbw\" (UID: \"82f75dd5-40cd-4cb3-9111-6213d6b43679\") " pod="openshift-image-registry/node-ca-gmbbw"
Oct 30 21:49:11.566538 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.565354    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/41fe6d63-9316-4916-b042-87ba7e602167-etc-openvswitch\") pod \"ovnkube-node-ml9xf\" (UID: \"41fe6d63-9316-4916-b042-87ba7e602167\") " pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf"
Oct 30 21:49:11.566538 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.565368    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d947d3db-ab6e-4874-8817-f8c74be138a1-lib-modules\") pod \"tuned-22pl2\" (UID: \"d947d3db-ab6e-4874-8817-f8c74be138a1\") " pod="openshift-cluster-node-tuning-operator/tuned-22pl2"
Oct 30 21:49:11.566538 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.565379    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/41fe6d63-9316-4916-b042-87ba7e602167-host-run-ovn-kubernetes\") pod \"ovnkube-node-ml9xf\" (UID: \"41fe6d63-9316-4916-b042-87ba7e602167\") " pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf"
Oct 30 21:49:11.566538 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.565404    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hghq\" (UniqueName: \"kubernetes.io/projected/41fe6d63-9316-4916-b042-87ba7e602167-kube-api-access-2hghq\") pod \"ovnkube-node-ml9xf\" (UID: \"41fe6d63-9316-4916-b042-87ba7e602167\") " pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf"
Oct 30 21:49:11.566538 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.565410    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5f468f3c-bf3d-437c-8ec8-a5d56f448d99-cni-binary-copy\") pod \"multus-p7zpx\" (UID: \"5f468f3c-bf3d-437c-8ec8-a5d56f448d99\") " pod="openshift-multus/multus-p7zpx"
Oct 30 21:49:11.566538 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.565437    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5f468f3c-bf3d-437c-8ec8-a5d56f448d99-multus-cni-dir\") pod \"multus-p7zpx\" (UID: \"5f468f3c-bf3d-437c-8ec8-a5d56f448d99\") " pod="openshift-multus/multus-p7zpx"
Oct 30 21:49:11.566538 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.565451    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/32d63b18-0fcb-4e4a-a24c-ee044dcb5a90-hosts-file\") pod \"node-resolver-gbpvx\" (UID: \"32d63b18-0fcb-4e4a-a24c-ee044dcb5a90\") " pod="openshift-dns/node-resolver-gbpvx"
Oct 30 21:49:11.566538 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.565460    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/41fe6d63-9316-4916-b042-87ba7e602167-etc-openvswitch\") pod \"ovnkube-node-ml9xf\" (UID: \"41fe6d63-9316-4916-b042-87ba7e602167\") " pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf"
Oct 30 21:49:11.566538 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.565483    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5f468f3c-bf3d-437c-8ec8-a5d56f448d99-multus-socket-dir-parent\") pod \"multus-p7zpx\" (UID: \"5f468f3c-bf3d-437c-8ec8-a5d56f448d99\") " pod="openshift-multus/multus-p7zpx"
Oct 30 21:49:11.566538 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.565483    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/924394f7-5b3b-4767-a1ff-f1d310ece4ed-socket-dir\") pod \"aws-ebs-csi-driver-node-lrc2c\" (UID: \"924394f7-5b3b-4767-a1ff-f1d310ece4ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lrc2c"
Oct 30 21:49:11.566538 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.565499    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/41fe6d63-9316-4916-b042-87ba7e602167-host-run-ovn-kubernetes\") pod \"ovnkube-node-ml9xf\" (UID: \"41fe6d63-9316-4916-b042-87ba7e602167\") " pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf"
Oct 30 21:49:11.566538 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.565555    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5f468f3c-bf3d-437c-8ec8-a5d56f448d99-multus-cni-dir\") pod \"multus-p7zpx\" (UID: \"5f468f3c-bf3d-437c-8ec8-a5d56f448d99\") " pod="openshift-multus/multus-p7zpx"
Oct 30 21:49:11.566538 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.565616    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d947d3db-ab6e-4874-8817-f8c74be138a1-etc-sysctl-d\") pod \"tuned-22pl2\" (UID: \"d947d3db-ab6e-4874-8817-f8c74be138a1\") " pod="openshift-cluster-node-tuning-operator/tuned-22pl2"
Oct 30 21:49:11.566538 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.565629    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5f468f3c-bf3d-437c-8ec8-a5d56f448d99-multus-socket-dir-parent\") pod \"multus-p7zpx\" (UID: \"5f468f3c-bf3d-437c-8ec8-a5d56f448d99\") " pod="openshift-multus/multus-p7zpx"
Oct 30 21:49:11.566538 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.565660    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkkmr\" (UniqueName: \"kubernetes.io/projected/d947d3db-ab6e-4874-8817-f8c74be138a1-kube-api-access-mkkmr\") pod \"tuned-22pl2\" (UID: \"d947d3db-ab6e-4874-8817-f8c74be138a1\") " pod="openshift-cluster-node-tuning-operator/tuned-22pl2"
Oct 30 21:49:11.566538 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.565686    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/41fe6d63-9316-4916-b042-87ba7e602167-run-ovn\") pod \"ovnkube-node-ml9xf\" (UID: \"41fe6d63-9316-4916-b042-87ba7e602167\") " pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf"
Oct 30 21:49:11.567441 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.565712    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7a6305de-8fd1-4d3e-9628-8c4cda07f379-cnibin\") pod \"multus-additional-cni-plugins-vrtnk\" (UID: \"7a6305de-8fd1-4d3e-9628-8c4cda07f379\") " pod="openshift-multus/multus-additional-cni-plugins-vrtnk"
Oct 30 21:49:11.567441 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.565740    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnt2l\" (UniqueName: \"kubernetes.io/projected/924394f7-5b3b-4767-a1ff-f1d310ece4ed-kube-api-access-qnt2l\") pod \"aws-ebs-csi-driver-node-lrc2c\" (UID: \"924394f7-5b3b-4767-a1ff-f1d310ece4ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lrc2c"
Oct 30 21:49:11.567441 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.565762    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/41fe6d63-9316-4916-b042-87ba7e602167-run-ovn\") pod \"ovnkube-node-ml9xf\" (UID: \"41fe6d63-9316-4916-b042-87ba7e602167\") " pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf"
Oct 30 21:49:11.567441 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.565766    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/41fe6d63-9316-4916-b042-87ba7e602167-host-run-netns\") pod \"ovnkube-node-ml9xf\" (UID: \"41fe6d63-9316-4916-b042-87ba7e602167\") " pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf"
Oct 30 21:49:11.567441 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.565778    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7a6305de-8fd1-4d3e-9628-8c4cda07f379-cnibin\") pod \"multus-additional-cni-plugins-vrtnk\" (UID: \"7a6305de-8fd1-4d3e-9628-8c4cda07f379\") " pod="openshift-multus/multus-additional-cni-plugins-vrtnk"
Oct 30 21:49:11.567441 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.565790    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/41fe6d63-9316-4916-b042-87ba7e602167-host-run-netns\") pod \"ovnkube-node-ml9xf\" (UID: \"41fe6d63-9316-4916-b042-87ba7e602167\") " pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf"
Oct 30 21:49:11.567441 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.565766    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d947d3db-ab6e-4874-8817-f8c74be138a1-etc-sysctl-d\") pod \"tuned-22pl2\" (UID: \"d947d3db-ab6e-4874-8817-f8c74be138a1\") " pod="openshift-cluster-node-tuning-operator/tuned-22pl2"
Oct 30 21:49:11.567441 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.565863    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/41fe6d63-9316-4916-b042-87ba7e602167-run-openvswitch\") pod \"ovnkube-node-ml9xf\" (UID: \"41fe6d63-9316-4916-b042-87ba7e602167\") " pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf"
Oct 30 21:49:11.567441 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.565887    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/41fe6d63-9316-4916-b042-87ba7e602167-ovnkube-config\") pod \"ovnkube-node-ml9xf\" (UID: \"41fe6d63-9316-4916-b042-87ba7e602167\") " pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf"
Oct 30 21:49:11.567441 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.565915    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/41fe6d63-9316-4916-b042-87ba7e602167-ovn-node-metrics-cert\") pod \"ovnkube-node-ml9xf\" (UID: \"41fe6d63-9316-4916-b042-87ba7e602167\") " pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf"
Oct 30 21:49:11.567441 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.565919    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/41fe6d63-9316-4916-b042-87ba7e602167-run-openvswitch\") pod \"ovnkube-node-ml9xf\" (UID: \"41fe6d63-9316-4916-b042-87ba7e602167\") " pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf"
Oct 30 21:49:11.567441 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.565941    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5f468f3c-bf3d-437c-8ec8-a5d56f448d99-host-run-k8s-cni-cncf-io\") pod \"multus-p7zpx\" (UID: \"5f468f3c-bf3d-437c-8ec8-a5d56f448d99\") " pod="openshift-multus/multus-p7zpx"
Oct 30 21:49:11.567441 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.565972    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5f468f3c-bf3d-437c-8ec8-a5d56f448d99-hostroot\") pod \"multus-p7zpx\" (UID: \"5f468f3c-bf3d-437c-8ec8-a5d56f448d99\") " pod="openshift-multus/multus-p7zpx"
Oct 30 21:49:11.567441 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.565994    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5f468f3c-bf3d-437c-8ec8-a5d56f448d99-host-run-k8s-cni-cncf-io\") pod \"multus-p7zpx\" (UID: \"5f468f3c-bf3d-437c-8ec8-a5d56f448d99\") " pod="openshift-multus/multus-p7zpx"
Oct 30 21:49:11.567441 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.566003    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b97qt\" (UniqueName: \"kubernetes.io/projected/82f75dd5-40cd-4cb3-9111-6213d6b43679-kube-api-access-b97qt\") pod \"node-ca-gmbbw\" (UID: \"82f75dd5-40cd-4cb3-9111-6213d6b43679\") " pod="openshift-image-registry/node-ca-gmbbw"
Oct 30 21:49:11.567441 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.566028    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5f468f3c-bf3d-437c-8ec8-a5d56f448d99-host-run-netns\") pod \"multus-p7zpx\" (UID: \"5f468f3c-bf3d-437c-8ec8-a5d56f448d99\") " pod="openshift-multus/multus-p7zpx"
Oct 30 21:49:11.567441 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.566038    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5f468f3c-bf3d-437c-8ec8-a5d56f448d99-hostroot\") pod \"multus-p7zpx\" (UID: \"5f468f3c-bf3d-437c-8ec8-a5d56f448d99\") " pod="openshift-multus/multus-p7zpx"
Oct 30 21:49:11.567441 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.566044    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5f468f3c-bf3d-437c-8ec8-a5d56f448d99-host-var-lib-kubelet\") pod \"multus-p7zpx\" (UID: \"5f468f3c-bf3d-437c-8ec8-a5d56f448d99\") " pod="openshift-multus/multus-p7zpx"
Oct 30 21:49:11.568068 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.566069    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5f468f3c-bf3d-437c-8ec8-a5d56f448d99-host-var-lib-kubelet\") pod \"multus-p7zpx\" (UID: \"5f468f3c-bf3d-437c-8ec8-a5d56f448d99\") " pod="openshift-multus/multus-p7zpx"
Oct 30 21:49:11.568068 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.566096    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5f468f3c-bf3d-437c-8ec8-a5d56f448d99-host-run-multus-certs\") pod \"multus-p7zpx\" (UID: \"5f468f3c-bf3d-437c-8ec8-a5d56f448d99\") " pod="openshift-multus/multus-p7zpx"
Oct 30 21:49:11.568068 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.566126    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5f468f3c-bf3d-437c-8ec8-a5d56f448d99-etc-kubernetes\") pod \"multus-p7zpx\" (UID: \"5f468f3c-bf3d-437c-8ec8-a5d56f448d99\") " pod="openshift-multus/multus-p7zpx"
Oct 30 21:49:11.568068 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.566155    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7a6305de-8fd1-4d3e-9628-8c4cda07f379-system-cni-dir\") pod \"multus-additional-cni-plugins-vrtnk\" (UID: \"7a6305de-8fd1-4d3e-9628-8c4cda07f379\") " pod="openshift-multus/multus-additional-cni-plugins-vrtnk"
Oct 30 21:49:11.568068 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.566168    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5f468f3c-bf3d-437c-8ec8-a5d56f448d99-host-run-netns\") pod \"multus-p7zpx\" (UID: \"5f468f3c-bf3d-437c-8ec8-a5d56f448d99\") " pod="openshift-multus/multus-p7zpx"
Oct 30 21:49:11.568068 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.566178    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5f468f3c-bf3d-437c-8ec8-a5d56f448d99-host-run-multus-certs\") pod \"multus-p7zpx\" (UID: \"5f468f3c-bf3d-437c-8ec8-a5d56f448d99\") " pod="openshift-multus/multus-p7zpx"
Oct 30 21:49:11.568068 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.566182    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7a6305de-8fd1-4d3e-9628-8c4cda07f379-cni-binary-copy\") pod \"multus-additional-cni-plugins-vrtnk\" (UID: \"7a6305de-8fd1-4d3e-9628-8c4cda07f379\") " pod="openshift-multus/multus-additional-cni-plugins-vrtnk"
Oct 30 21:49:11.568068 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.566195    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5f468f3c-bf3d-437c-8ec8-a5d56f448d99-etc-kubernetes\") pod \"multus-p7zpx\" (UID: \"5f468f3c-bf3d-437c-8ec8-a5d56f448d99\") " pod="openshift-multus/multus-p7zpx"
Oct 30 21:49:11.568068 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.566212    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d947d3db-ab6e-4874-8817-f8c74be138a1-etc-modprobe-d\") pod \"tuned-22pl2\" (UID: \"d947d3db-ab6e-4874-8817-f8c74be138a1\") " pod="openshift-cluster-node-tuning-operator/tuned-22pl2"
Oct 30 21:49:11.568068 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.566220    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7a6305de-8fd1-4d3e-9628-8c4cda07f379-system-cni-dir\") pod \"multus-additional-cni-plugins-vrtnk\" (UID: \"7a6305de-8fd1-4d3e-9628-8c4cda07f379\") " pod="openshift-multus/multus-additional-cni-plugins-vrtnk"
Oct 30 21:49:11.568068 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.566250    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d947d3db-ab6e-4874-8817-f8c74be138a1-host\") pod \"tuned-22pl2\" (UID: \"d947d3db-ab6e-4874-8817-f8c74be138a1\") " pod="openshift-cluster-node-tuning-operator/tuned-22pl2"
Oct 30 21:49:11.568068 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.566289    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czfxj\" (UniqueName: \"kubernetes.io/projected/a6980f40-89cf-4f3d-8cd5-91969ae5a276-kube-api-access-czfxj\") pod \"network-metrics-daemon-xbh9m\" (UID: \"a6980f40-89cf-4f3d-8cd5-91969ae5a276\") " pod="openshift-multus/network-metrics-daemon-xbh9m"
Oct 30 21:49:11.568068 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.566305    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d947d3db-ab6e-4874-8817-f8c74be138a1-host\") pod \"tuned-22pl2\" (UID: \"d947d3db-ab6e-4874-8817-f8c74be138a1\") " pod="openshift-cluster-node-tuning-operator/tuned-22pl2"
Oct 30 21:49:11.568068 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.566318    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/41fe6d63-9316-4916-b042-87ba7e602167-host-cni-bin\") pod \"ovnkube-node-ml9xf\" (UID: \"41fe6d63-9316-4916-b042-87ba7e602167\") " pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf"
Oct 30 21:49:11.568068 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.566321    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d947d3db-ab6e-4874-8817-f8c74be138a1-etc-modprobe-d\") pod \"tuned-22pl2\" (UID: \"d947d3db-ab6e-4874-8817-f8c74be138a1\") " pod="openshift-cluster-node-tuning-operator/tuned-22pl2"
Oct 30 21:49:11.568068 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.566345    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/41fe6d63-9316-4916-b042-87ba7e602167-ovnkube-script-lib\") pod \"ovnkube-node-ml9xf\" (UID: \"41fe6d63-9316-4916-b042-87ba7e602167\") " pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf"
Oct 30 21:49:11.568068 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.566381    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/41fe6d63-9316-4916-b042-87ba7e602167-host-cni-bin\") pod \"ovnkube-node-ml9xf\" (UID: \"41fe6d63-9316-4916-b042-87ba7e602167\") " pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf"
Oct 30 21:49:11.568068 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.566398    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0ad59239-edcc-46a4-a5a1-1debb63e3a5d-konnectivity-ca\") pod \"konnectivity-agent-w72c9\" (UID: \"0ad59239-edcc-46a4-a5a1-1debb63e3a5d\") " pod="kube-system/konnectivity-agent-w72c9"
Oct 30 21:49:11.568946 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.566425    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/924394f7-5b3b-4767-a1ff-f1d310ece4ed-sys-fs\") pod \"aws-ebs-csi-driver-node-lrc2c\" (UID: \"924394f7-5b3b-4767-a1ff-f1d310ece4ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lrc2c"
Oct 30 21:49:11.568946 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.566498    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d947d3db-ab6e-4874-8817-f8c74be138a1-etc-systemd\") pod \"tuned-22pl2\" (UID: \"d947d3db-ab6e-4874-8817-f8c74be138a1\") " pod="openshift-cluster-node-tuning-operator/tuned-22pl2"
Oct 30 21:49:11.568946 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.566515    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d947d3db-ab6e-4874-8817-f8c74be138a1-tmp\") pod \"tuned-22pl2\" (UID: \"d947d3db-ab6e-4874-8817-f8c74be138a1\") " pod="openshift-cluster-node-tuning-operator/tuned-22pl2"
Oct 30 21:49:11.568946 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.566535    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxd2f\" (UniqueName: \"kubernetes.io/projected/32d63b18-0fcb-4e4a-a24c-ee044dcb5a90-kube-api-access-pxd2f\") pod \"node-resolver-gbpvx\" (UID: \"32d63b18-0fcb-4e4a-a24c-ee044dcb5a90\") " pod="openshift-dns/node-resolver-gbpvx"
Oct 30 21:49:11.568946 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.566557    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/924394f7-5b3b-4767-a1ff-f1d310ece4ed-registration-dir\") pod \"aws-ebs-csi-driver-node-lrc2c\" (UID: \"924394f7-5b3b-4767-a1ff-f1d310ece4ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lrc2c"
Oct 30 21:49:11.568946 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.566573    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5f468f3c-bf3d-437c-8ec8-a5d56f448d99-system-cni-dir\") pod \"multus-p7zpx\" (UID: \"5f468f3c-bf3d-437c-8ec8-a5d56f448d99\") " pod="openshift-multus/multus-p7zpx"
Oct 30 21:49:11.568946 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.566592    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8w8vm\" (UniqueName: \"kubernetes.io/projected/f22e4514-ac6d-481d-8afe-6e2e4cdc3aa1-kube-api-access-8w8vm\") pod \"iptables-alerter-t24lz\" (UID: \"f22e4514-ac6d-481d-8afe-6e2e4cdc3aa1\") " pod="openshift-network-operator/iptables-alerter-t24lz"
Oct 30 21:49:11.568946 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.566609    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32d63b18-0fcb-4e4a-a24c-ee044dcb5a90-tmp-dir\") pod \"node-resolver-gbpvx\" (UID: \"32d63b18-0fcb-4e4a-a24c-ee044dcb5a90\") " pod="openshift-dns/node-resolver-gbpvx"
Oct 30 21:49:11.568946 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.566626    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b954bab2-9dd7-4b39-bda6-3fec87381aff-kubelet-config\") pod \"global-pull-secret-syncer-cck7d\" (UID: \"b954bab2-9dd7-4b39-bda6-3fec87381aff\") " pod="kube-system/global-pull-secret-syncer-cck7d"
Oct 30 21:49:11.568946 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.566650    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b954bab2-9dd7-4b39-bda6-3fec87381aff-dbus\") pod \"global-pull-secret-syncer-cck7d\" (UID: \"b954bab2-9dd7-4b39-bda6-3fec87381aff\") " pod="kube-system/global-pull-secret-syncer-cck7d"
Oct 30 21:49:11.568946 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.566660    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/924394f7-5b3b-4767-a1ff-f1d310ece4ed-sys-fs\") pod \"aws-ebs-csi-driver-node-lrc2c\" (UID: \"924394f7-5b3b-4767-a1ff-f1d310ece4ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lrc2c"
Oct 30 21:49:11.568946 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.566605    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d947d3db-ab6e-4874-8817-f8c74be138a1-etc-systemd\") pod \"tuned-22pl2\" (UID: \"d947d3db-ab6e-4874-8817-f8c74be138a1\") " pod="openshift-cluster-node-tuning-operator/tuned-22pl2"
Oct 30 21:49:11.568946 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.566684    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5f468f3c-bf3d-437c-8ec8-a5d56f448d99-system-cni-dir\") pod \"multus-p7zpx\" (UID: \"5f468f3c-bf3d-437c-8ec8-a5d56f448d99\") " pod="openshift-multus/multus-p7zpx"
Oct 30 21:49:11.568946 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.566666    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7a6305de-8fd1-4d3e-9628-8c4cda07f379-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vrtnk\" (UID: \"7a6305de-8fd1-4d3e-9628-8c4cda07f379\") " pod="openshift-multus/multus-additional-cni-plugins-vrtnk"
Oct 30 21:49:11.568946 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.566796    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5f468f3c-bf3d-437c-8ec8-a5d56f448d99-multus-daemon-config\") pod \"multus-p7zpx\" (UID: \"5f468f3c-bf3d-437c-8ec8-a5d56f448d99\") " pod="openshift-multus/multus-p7zpx"
Oct 30 21:49:11.568946 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.566890    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b954bab2-9dd7-4b39-bda6-3fec87381aff-dbus\") pod \"global-pull-secret-syncer-cck7d\" (UID: \"b954bab2-9dd7-4b39-bda6-3fec87381aff\") " pod="kube-system/global-pull-secret-syncer-cck7d"
Oct 30 21:49:11.568946 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.566984    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/924394f7-5b3b-4767-a1ff-f1d310ece4ed-registration-dir\") pod \"aws-ebs-csi-driver-node-lrc2c\" (UID: \"924394f7-5b3b-4767-a1ff-f1d310ece4ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lrc2c"
Oct 30 21:49:11.569822 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.567040    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b954bab2-9dd7-4b39-bda6-3fec87381aff-kubelet-config\") pod \"global-pull-secret-syncer-cck7d\" (UID: \"b954bab2-9dd7-4b39-bda6-3fec87381aff\") " pod="kube-system/global-pull-secret-syncer-cck7d"
Oct 30 21:49:11.569822 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.567091    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f22e4514-ac6d-481d-8afe-6e2e4cdc3aa1-iptables-alerter-script\") pod \"iptables-alerter-t24lz\" (UID: \"f22e4514-ac6d-481d-8afe-6e2e4cdc3aa1\") " pod="openshift-network-operator/iptables-alerter-t24lz"
Oct 30 21:49:11.569822 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.567148    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32d63b18-0fcb-4e4a-a24c-ee044dcb5a90-tmp-dir\") pod \"node-resolver-gbpvx\" (UID: \"32d63b18-0fcb-4e4a-a24c-ee044dcb5a90\") " pod="openshift-dns/node-resolver-gbpvx"
Oct 30 21:49:11.569822 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.567151    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/924394f7-5b3b-4767-a1ff-f1d310ece4ed-device-dir\") pod \"aws-ebs-csi-driver-node-lrc2c\" (UID: \"924394f7-5b3b-4767-a1ff-f1d310ece4ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lrc2c"
Oct 30 21:49:11.569822 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.567185    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d947d3db-ab6e-4874-8817-f8c74be138a1-etc-sysctl-conf\") pod \"tuned-22pl2\" (UID: \"d947d3db-ab6e-4874-8817-f8c74be138a1\") " pod="openshift-cluster-node-tuning-operator/tuned-22pl2"
Oct 30 21:49:11.569822 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.567213    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/924394f7-5b3b-4767-a1ff-f1d310ece4ed-device-dir\") pod \"aws-ebs-csi-driver-node-lrc2c\" (UID: \"924394f7-5b3b-4767-a1ff-f1d310ece4ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lrc2c"
Oct 30 21:49:11.569822 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.567244    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7a6305de-8fd1-4d3e-9628-8c4cda07f379-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vrtnk\" (UID: \"7a6305de-8fd1-4d3e-9628-8c4cda07f379\") " pod="openshift-multus/multus-additional-cni-plugins-vrtnk"
Oct 30 21:49:11.569822 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.567253    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/82f75dd5-40cd-4cb3-9111-6213d6b43679-serviceca\") pod \"node-ca-gmbbw\" (UID: \"82f75dd5-40cd-4cb3-9111-6213d6b43679\") " pod="openshift-image-registry/node-ca-gmbbw"
Oct 30 21:49:11.569822 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.567253    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d947d3db-ab6e-4874-8817-f8c74be138a1-var-lib-kubelet\") pod \"tuned-22pl2\" (UID: \"d947d3db-ab6e-4874-8817-f8c74be138a1\") " pod="openshift-cluster-node-tuning-operator/tuned-22pl2"
Oct 30 21:49:11.569822 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.567307    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7a6305de-8fd1-4d3e-9628-8c4cda07f379-cni-binary-copy\") pod \"multus-additional-cni-plugins-vrtnk\" (UID: \"7a6305de-8fd1-4d3e-9628-8c4cda07f379\") " pod="openshift-multus/multus-additional-cni-plugins-vrtnk"
Oct 30 21:49:11.569822 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.567301    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/41fe6d63-9316-4916-b042-87ba7e602167-node-log\") pod \"ovnkube-node-ml9xf\" (UID: \"41fe6d63-9316-4916-b042-87ba7e602167\") " pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf"
Oct 30 21:49:11.569822 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.567345    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/41fe6d63-9316-4916-b042-87ba7e602167-host-cni-netd\") pod \"ovnkube-node-ml9xf\" (UID: \"41fe6d63-9316-4916-b042-87ba7e602167\") " pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf"
Oct 30 21:49:11.569822 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.567386    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/41fe6d63-9316-4916-b042-87ba7e602167-host-cni-netd\") pod \"ovnkube-node-ml9xf\" (UID: \"41fe6d63-9316-4916-b042-87ba7e602167\") " pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf"
Oct 30 21:49:11.569822 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.567394    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d947d3db-ab6e-4874-8817-f8c74be138a1-etc-sysctl-conf\") pod \"tuned-22pl2\" (UID: \"d947d3db-ab6e-4874-8817-f8c74be138a1\") " pod="openshift-cluster-node-tuning-operator/tuned-22pl2"
Oct 30 21:49:11.569822 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.567346    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/41fe6d63-9316-4916-b042-87ba7e602167-node-log\") pod \"ovnkube-node-ml9xf\" (UID: \"41fe6d63-9316-4916-b042-87ba7e602167\") " pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf"
Oct 30 21:49:11.569822 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.567450    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d947d3db-ab6e-4874-8817-f8c74be138a1-var-lib-kubelet\") pod \"tuned-22pl2\" (UID: \"d947d3db-ab6e-4874-8817-f8c74be138a1\") " pod="openshift-cluster-node-tuning-operator/tuned-22pl2"
Oct 30 21:49:11.569822 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.567636    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7a6305de-8fd1-4d3e-9628-8c4cda07f379-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vrtnk\" (UID: \"7a6305de-8fd1-4d3e-9628-8c4cda07f379\") " pod="openshift-multus/multus-additional-cni-plugins-vrtnk"
Oct 30 21:49:11.570702 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.567732    2573 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 21:49:11.570702 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.567954    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/41fe6d63-9316-4916-b042-87ba7e602167-ovnkube-config\") pod \"ovnkube-node-ml9xf\" (UID: \"41fe6d63-9316-4916-b042-87ba7e602167\") " pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf"
Oct 30 21:49:11.570702 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.568008    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0ad59239-edcc-46a4-a5a1-1debb63e3a5d-konnectivity-ca\") pod \"konnectivity-agent-w72c9\" (UID: \"0ad59239-edcc-46a4-a5a1-1debb63e3a5d\") " pod="kube-system/konnectivity-agent-w72c9"
Oct 30 21:49:11.570702 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.568365    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/41fe6d63-9316-4916-b042-87ba7e602167-ovnkube-script-lib\") pod \"ovnkube-node-ml9xf\" (UID: \"41fe6d63-9316-4916-b042-87ba7e602167\") " pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf"
Oct 30 21:49:11.570884 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.570859    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d947d3db-ab6e-4874-8817-f8c74be138a1-tmp\") pod \"tuned-22pl2\" (UID: \"d947d3db-ab6e-4874-8817-f8c74be138a1\") " pod="openshift-cluster-node-tuning-operator/tuned-22pl2"
Oct 30 21:49:11.571041 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.571021    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d947d3db-ab6e-4874-8817-f8c74be138a1-etc-tuned\") pod \"tuned-22pl2\" (UID: \"d947d3db-ab6e-4874-8817-f8c74be138a1\") " pod="openshift-cluster-node-tuning-operator/tuned-22pl2"
Oct 30 21:49:11.571165 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.571145    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/41fe6d63-9316-4916-b042-87ba7e602167-ovn-node-metrics-cert\") pod \"ovnkube-node-ml9xf\" (UID: \"41fe6d63-9316-4916-b042-87ba7e602167\") " pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf"
Oct 30 21:49:11.571365 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.571351    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0ad59239-edcc-46a4-a5a1-1debb63e3a5d-agent-certs\") pod \"konnectivity-agent-w72c9\" (UID: \"0ad59239-edcc-46a4-a5a1-1debb63e3a5d\") " pod="kube-system/konnectivity-agent-w72c9"
Oct 30 21:49:11.624880 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:11.624856    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 21:49:11.624880 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:11.624882    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 21:49:11.625022 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:11.624894    2573 projected.go:196] Error preparing data for projected volume kube-api-access-j6x28 for pod openshift-network-diagnostics/network-check-target-w9tx8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:49:11.625022 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:11.624954    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/55619a2e-f6e9-4719-b362-30b69d553967-kube-api-access-j6x28 podName:55619a2e-f6e9-4719-b362-30b69d553967 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:12.124938913 +0000 UTC m=+3.247563634 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-j6x28" (UniqueName: "kubernetes.io/projected/55619a2e-f6e9-4719-b362-30b69d553967-kube-api-access-j6x28") pod "network-check-target-w9tx8" (UID: "55619a2e-f6e9-4719-b362-30b69d553967") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:49:11.628269 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.628238    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwvcm\" (UniqueName: \"kubernetes.io/projected/7a6305de-8fd1-4d3e-9628-8c4cda07f379-kube-api-access-lwvcm\") pod \"multus-additional-cni-plugins-vrtnk\" (UID: \"7a6305de-8fd1-4d3e-9628-8c4cda07f379\") " pod="openshift-multus/multus-additional-cni-plugins-vrtnk"
Oct 30 21:49:11.628269 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.628258    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkkmr\" (UniqueName: \"kubernetes.io/projected/d947d3db-ab6e-4874-8817-f8c74be138a1-kube-api-access-mkkmr\") pod \"tuned-22pl2\" (UID: \"d947d3db-ab6e-4874-8817-f8c74be138a1\") " pod="openshift-cluster-node-tuning-operator/tuned-22pl2"
Oct 30 21:49:11.628620 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.628599    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgn7x\" (UniqueName: \"kubernetes.io/projected/5f468f3c-bf3d-437c-8ec8-a5d56f448d99-kube-api-access-rgn7x\") pod \"multus-p7zpx\" (UID: \"5f468f3c-bf3d-437c-8ec8-a5d56f448d99\") " pod="openshift-multus/multus-p7zpx"
Oct 30 21:49:11.628659 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.628606    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hghq\" (UniqueName: \"kubernetes.io/projected/41fe6d63-9316-4916-b042-87ba7e602167-kube-api-access-2hghq\") pod \"ovnkube-node-ml9xf\" (UID: \"41fe6d63-9316-4916-b042-87ba7e602167\") " pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf"
Oct 30 21:49:11.628958 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.628941    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b97qt\" (UniqueName: \"kubernetes.io/projected/82f75dd5-40cd-4cb3-9111-6213d6b43679-kube-api-access-b97qt\") pod \"node-ca-gmbbw\" (UID: \"82f75dd5-40cd-4cb3-9111-6213d6b43679\") " pod="openshift-image-registry/node-ca-gmbbw"
Oct 30 21:49:11.632753 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.632727    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnt2l\" (UniqueName: \"kubernetes.io/projected/924394f7-5b3b-4767-a1ff-f1d310ece4ed-kube-api-access-qnt2l\") pod \"aws-ebs-csi-driver-node-lrc2c\" (UID: \"924394f7-5b3b-4767-a1ff-f1d310ece4ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lrc2c"
Oct 30 21:49:11.634121 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.634101    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxd2f\" (UniqueName: \"kubernetes.io/projected/32d63b18-0fcb-4e4a-a24c-ee044dcb5a90-kube-api-access-pxd2f\") pod \"node-resolver-gbpvx\" (UID: \"32d63b18-0fcb-4e4a-a24c-ee044dcb5a90\") " pod="openshift-dns/node-resolver-gbpvx"
Oct 30 21:49:11.634249 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.634226    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czfxj\" (UniqueName: \"kubernetes.io/projected/a6980f40-89cf-4f3d-8cd5-91969ae5a276-kube-api-access-czfxj\") pod \"network-metrics-daemon-xbh9m\" (UID: \"a6980f40-89cf-4f3d-8cd5-91969ae5a276\") " pod="openshift-multus/network-metrics-daemon-xbh9m"
Oct 30 21:49:11.668376 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.668350    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8w8vm\" (UniqueName: \"kubernetes.io/projected/f22e4514-ac6d-481d-8afe-6e2e4cdc3aa1-kube-api-access-8w8vm\") pod \"iptables-alerter-t24lz\" (UID: \"f22e4514-ac6d-481d-8afe-6e2e4cdc3aa1\") " pod="openshift-network-operator/iptables-alerter-t24lz"
Oct 30 21:49:11.668494 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.668387    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f22e4514-ac6d-481d-8afe-6e2e4cdc3aa1-iptables-alerter-script\") pod \"iptables-alerter-t24lz\" (UID: \"f22e4514-ac6d-481d-8afe-6e2e4cdc3aa1\") " pod="openshift-network-operator/iptables-alerter-t24lz"
Oct 30 21:49:11.668494 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.668457    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f22e4514-ac6d-481d-8afe-6e2e4cdc3aa1-host-slash\") pod \"iptables-alerter-t24lz\" (UID: \"f22e4514-ac6d-481d-8afe-6e2e4cdc3aa1\") " pod="openshift-network-operator/iptables-alerter-t24lz"
Oct 30 21:49:11.668568 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.668530    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f22e4514-ac6d-481d-8afe-6e2e4cdc3aa1-host-slash\") pod \"iptables-alerter-t24lz\" (UID: \"f22e4514-ac6d-481d-8afe-6e2e4cdc3aa1\") " pod="openshift-network-operator/iptables-alerter-t24lz"
Oct 30 21:49:11.668988 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.668969    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f22e4514-ac6d-481d-8afe-6e2e4cdc3aa1-iptables-alerter-script\") pod \"iptables-alerter-t24lz\" (UID: \"f22e4514-ac6d-481d-8afe-6e2e4cdc3aa1\") " pod="openshift-network-operator/iptables-alerter-t24lz"
Oct 30 21:49:11.731758 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.731723    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8w8vm\" (UniqueName: \"kubernetes.io/projected/f22e4514-ac6d-481d-8afe-6e2e4cdc3aa1-kube-api-access-8w8vm\") pod \"iptables-alerter-t24lz\" (UID: \"f22e4514-ac6d-481d-8afe-6e2e4cdc3aa1\") " pod="openshift-network-operator/iptables-alerter-t24lz"
Oct 30 21:49:11.748843 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.748823    2573 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:49:11.750019 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.750003    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-w72c9"
Oct 30 21:49:11.751192 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.751177    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf"
Oct 30 21:49:11.753256 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.753244    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gbpvx"
Oct 30 21:49:11.760413 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.760392    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lrc2c"
Oct 30 21:49:11.761494 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:11.761467    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ad59239_edcc_46a4_a5a1_1debb63e3a5d.slice/crio-11dac6cc6798646aadb4d4e9dc28993e017dd18a9f1a636bf8789dce74026edd WatchSource:0}: Error finding container 11dac6cc6798646aadb4d4e9dc28993e017dd18a9f1a636bf8789dce74026edd: Status 404 returned error can't find the container with id 11dac6cc6798646aadb4d4e9dc28993e017dd18a9f1a636bf8789dce74026edd
Oct 30 21:49:11.763523 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.763503    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-22pl2"
Oct 30 21:49:11.770561 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.770535    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-p7zpx"
Oct 30 21:49:11.771341 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:11.771317    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32d63b18_0fcb_4e4a_a24c_ee044dcb5a90.slice/crio-f81df74784eeabd8abcee1495fda7eb479bb06f0826ce1b438d3632d3bd104b4 WatchSource:0}: Error finding container f81df74784eeabd8abcee1495fda7eb479bb06f0826ce1b438d3632d3bd104b4: Status 404 returned error can't find the container with id f81df74784eeabd8abcee1495fda7eb479bb06f0826ce1b438d3632d3bd104b4
Oct 30 21:49:11.771990 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:11.771963    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod924394f7_5b3b_4767_a1ff_f1d310ece4ed.slice/crio-5b7fa7c8959b27a9ea4f5af1c51cb119eefb9044902ab5949a2f83aa030ae7cb WatchSource:0}: Error finding container 5b7fa7c8959b27a9ea4f5af1c51cb119eefb9044902ab5949a2f83aa030ae7cb: Status 404 returned error can't find the container with id 5b7fa7c8959b27a9ea4f5af1c51cb119eefb9044902ab5949a2f83aa030ae7cb
Oct 30 21:49:11.775418 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:11.775390    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd947d3db_ab6e_4874_8817_f8c74be138a1.slice/crio-2aefc6730b109bc0c1e053241ab815ac9def35808ecdbbcdb06989fb5e26b585 WatchSource:0}: Error finding container 2aefc6730b109bc0c1e053241ab815ac9def35808ecdbbcdb06989fb5e26b585: Status 404 returned error can't find the container with id 2aefc6730b109bc0c1e053241ab815ac9def35808ecdbbcdb06989fb5e26b585
Oct 30 21:49:11.779872 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:11.779855    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f468f3c_bf3d_437c_8ec8_a5d56f448d99.slice/crio-06de6ffcc0e4da71a6468cd1841f7fb4b2018aedf613cdaed926b55ac3d8b6fe WatchSource:0}: Error finding container 06de6ffcc0e4da71a6468cd1841f7fb4b2018aedf613cdaed926b55ac3d8b6fe: Status 404 returned error can't find the container with id 06de6ffcc0e4da71a6468cd1841f7fb4b2018aedf613cdaed926b55ac3d8b6fe
Oct 30 21:49:11.785001 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.784985    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-vrtnk"
Oct 30 21:49:11.787204 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.787187    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-gmbbw"
Oct 30 21:49:11.788724 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.788710    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-t24lz"
Oct 30 21:49:11.793305 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:11.793280    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a6305de_8fd1_4d3e_9628_8c4cda07f379.slice/crio-f6a186b19f8e7b056b1e01d0d6141ef6ea402c1dbb9a2d946a89669c6b60f63f WatchSource:0}: Error finding container f6a186b19f8e7b056b1e01d0d6141ef6ea402c1dbb9a2d946a89669c6b60f63f: Status 404 returned error can't find the container with id f6a186b19f8e7b056b1e01d0d6141ef6ea402c1dbb9a2d946a89669c6b60f63f
Oct 30 21:49:11.796540 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:11.796483    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82f75dd5_40cd_4cb3_9111_6213d6b43679.slice/crio-48c8198c2f2b74b17130ae9678885f3f7d4c981c4bc55458dc71962684f9b0b8 WatchSource:0}: Error finding container 48c8198c2f2b74b17130ae9678885f3f7d4c981c4bc55458dc71962684f9b0b8: Status 404 returned error can't find the container with id 48c8198c2f2b74b17130ae9678885f3f7d4c981c4bc55458dc71962684f9b0b8
Oct 30 21:49:11.799173 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:11.799152    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf22e4514_ac6d_481d_8afe_6e2e4cdc3aa1.slice/crio-ff7b45b007ad888b3545c4a198046fb18b188e5a1e0e7854aa6c5a60dc784808 WatchSource:0}: Error finding container ff7b45b007ad888b3545c4a198046fb18b188e5a1e0e7854aa6c5a60dc784808: Status 404 returned error can't find the container with id ff7b45b007ad888b3545c4a198046fb18b188e5a1e0e7854aa6c5a60dc784808
Oct 30 21:49:11.838476 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:11.838430    2573 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:49:12.071773 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:12.071731    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b954bab2-9dd7-4b39-bda6-3fec87381aff-original-pull-secret\") pod \"global-pull-secret-syncer-cck7d\" (UID: \"b954bab2-9dd7-4b39-bda6-3fec87381aff\") " pod="kube-system/global-pull-secret-syncer-cck7d"
Oct 30 21:49:12.071967 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:12.071800    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6980f40-89cf-4f3d-8cd5-91969ae5a276-metrics-certs\") pod \"network-metrics-daemon-xbh9m\" (UID: \"a6980f40-89cf-4f3d-8cd5-91969ae5a276\") " pod="openshift-multus/network-metrics-daemon-xbh9m"
Oct 30 21:49:12.071967 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:12.071842    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 21:49:12.071967 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:12.071908    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b954bab2-9dd7-4b39-bda6-3fec87381aff-original-pull-secret podName:b954bab2-9dd7-4b39-bda6-3fec87381aff nodeName:}" failed. No retries permitted until 2026-10-30 21:49:13.071892681 +0000 UTC m=+4.194517390 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b954bab2-9dd7-4b39-bda6-3fec87381aff-original-pull-secret") pod "global-pull-secret-syncer-cck7d" (UID: "b954bab2-9dd7-4b39-bda6-3fec87381aff") : object "kube-system"/"original-pull-secret" not registered
Oct 30 21:49:12.071967 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:12.071939    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:49:12.072173 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:12.071996    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6980f40-89cf-4f3d-8cd5-91969ae5a276-metrics-certs podName:a6980f40-89cf-4f3d-8cd5-91969ae5a276 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:13.071981136 +0000 UTC m=+4.194605844 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a6980f40-89cf-4f3d-8cd5-91969ae5a276-metrics-certs") pod "network-metrics-daemon-xbh9m" (UID: "a6980f40-89cf-4f3d-8cd5-91969ae5a276") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:49:12.173899 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:12.173814    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6x28\" (UniqueName: \"kubernetes.io/projected/55619a2e-f6e9-4719-b362-30b69d553967-kube-api-access-j6x28\") pod \"network-check-target-w9tx8\" (UID: \"55619a2e-f6e9-4719-b362-30b69d553967\") " pod="openshift-network-diagnostics/network-check-target-w9tx8"
Oct 30 21:49:12.174045 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:12.173974    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 21:49:12.174045 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:12.173998    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 21:49:12.174045 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:12.174035    2573 projected.go:196] Error preparing data for projected volume kube-api-access-j6x28 for pod openshift-network-diagnostics/network-check-target-w9tx8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:49:12.174208 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:12.174150    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/55619a2e-f6e9-4719-b362-30b69d553967-kube-api-access-j6x28 podName:55619a2e-f6e9-4719-b362-30b69d553967 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:13.174082398 +0000 UTC m=+4.296707120 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-j6x28" (UniqueName: "kubernetes.io/projected/55619a2e-f6e9-4719-b362-30b69d553967-kube-api-access-j6x28") pod "network-check-target-w9tx8" (UID: "55619a2e-f6e9-4719-b362-30b69d553967") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:49:12.489964 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:12.489878    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gbpvx" event={"ID":"32d63b18-0fcb-4e4a-a24c-ee044dcb5a90","Type":"ContainerStarted","Data":"f81df74784eeabd8abcee1495fda7eb479bb06f0826ce1b438d3632d3bd104b4"}
Oct 30 21:49:12.491359 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:12.491328    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-w72c9" event={"ID":"0ad59239-edcc-46a4-a5a1-1debb63e3a5d","Type":"ContainerStarted","Data":"11dac6cc6798646aadb4d4e9dc28993e017dd18a9f1a636bf8789dce74026edd"}
Oct 30 21:49:12.492635 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:12.492605    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vrtnk" event={"ID":"7a6305de-8fd1-4d3e-9628-8c4cda07f379","Type":"ContainerStarted","Data":"f6a186b19f8e7b056b1e01d0d6141ef6ea402c1dbb9a2d946a89669c6b60f63f"}
Oct 30 21:49:12.493804 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:12.493781    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lrc2c" event={"ID":"924394f7-5b3b-4767-a1ff-f1d310ece4ed","Type":"ContainerStarted","Data":"5b7fa7c8959b27a9ea4f5af1c51cb119eefb9044902ab5949a2f83aa030ae7cb"}
Oct 30 21:49:12.495245 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:12.495218    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf" event={"ID":"41fe6d63-9316-4916-b042-87ba7e602167","Type":"ContainerStarted","Data":"ef9db6f7956a084b994e446966ee810675c93075b4589ea22626894079d461b1"}
Oct 30 21:49:12.498065 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:12.498041    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-t24lz" event={"ID":"f22e4514-ac6d-481d-8afe-6e2e4cdc3aa1","Type":"ContainerStarted","Data":"ff7b45b007ad888b3545c4a198046fb18b188e5a1e0e7854aa6c5a60dc784808"}
Oct 30 21:49:12.499844 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:12.499821    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-gmbbw" event={"ID":"82f75dd5-40cd-4cb3-9111-6213d6b43679","Type":"ContainerStarted","Data":"48c8198c2f2b74b17130ae9678885f3f7d4c981c4bc55458dc71962684f9b0b8"}
Oct 30 21:49:12.501282 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:12.501257    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-p7zpx" event={"ID":"5f468f3c-bf3d-437c-8ec8-a5d56f448d99","Type":"ContainerStarted","Data":"06de6ffcc0e4da71a6468cd1841f7fb4b2018aedf613cdaed926b55ac3d8b6fe"}
Oct 30 21:49:12.502572 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:12.502547    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-22pl2" event={"ID":"d947d3db-ab6e-4874-8817-f8c74be138a1","Type":"ContainerStarted","Data":"2aefc6730b109bc0c1e053241ab815ac9def35808ecdbbcdb06989fb5e26b585"}
Oct 30 21:49:12.536475 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:12.536398    2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 21:44:11 +0000 UTC" deadline="2028-04-28 21:18:11.387665597 +0000 UTC"
Oct 30 21:49:12.536475 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:12.536474    2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13103h28m58.851195841s"
Oct 30 21:49:13.082920 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:13.082876    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b954bab2-9dd7-4b39-bda6-3fec87381aff-original-pull-secret\") pod \"global-pull-secret-syncer-cck7d\" (UID: \"b954bab2-9dd7-4b39-bda6-3fec87381aff\") " pod="kube-system/global-pull-secret-syncer-cck7d"
Oct 30 21:49:13.083111 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:13.082954    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6980f40-89cf-4f3d-8cd5-91969ae5a276-metrics-certs\") pod \"network-metrics-daemon-xbh9m\" (UID: \"a6980f40-89cf-4f3d-8cd5-91969ae5a276\") " pod="openshift-multus/network-metrics-daemon-xbh9m"
Oct 30 21:49:13.083193 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:13.083119    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:49:13.083315 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:13.083285    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6980f40-89cf-4f3d-8cd5-91969ae5a276-metrics-certs podName:a6980f40-89cf-4f3d-8cd5-91969ae5a276 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:15.08325856 +0000 UTC m=+6.205883283 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a6980f40-89cf-4f3d-8cd5-91969ae5a276-metrics-certs") pod "network-metrics-daemon-xbh9m" (UID: "a6980f40-89cf-4f3d-8cd5-91969ae5a276") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:49:13.083445 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:13.083374    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 21:49:13.083445 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:13.083411    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b954bab2-9dd7-4b39-bda6-3fec87381aff-original-pull-secret podName:b954bab2-9dd7-4b39-bda6-3fec87381aff nodeName:}" failed. No retries permitted until 2026-10-30 21:49:15.083399294 +0000 UTC m=+6.206024006 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b954bab2-9dd7-4b39-bda6-3fec87381aff-original-pull-secret") pod "global-pull-secret-syncer-cck7d" (UID: "b954bab2-9dd7-4b39-bda6-3fec87381aff") : object "kube-system"/"original-pull-secret" not registered
Oct 30 21:49:13.183833 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:13.183789    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6x28\" (UniqueName: \"kubernetes.io/projected/55619a2e-f6e9-4719-b362-30b69d553967-kube-api-access-j6x28\") pod \"network-check-target-w9tx8\" (UID: \"55619a2e-f6e9-4719-b362-30b69d553967\") " pod="openshift-network-diagnostics/network-check-target-w9tx8"
Oct 30 21:49:13.184032 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:13.183988    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 21:49:13.184032 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:13.184013    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 21:49:13.184032 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:13.184027    2573 projected.go:196] Error preparing data for projected volume kube-api-access-j6x28 for pod openshift-network-diagnostics/network-check-target-w9tx8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:49:13.184192 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:13.184091    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/55619a2e-f6e9-4719-b362-30b69d553967-kube-api-access-j6x28 podName:55619a2e-f6e9-4719-b362-30b69d553967 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:15.184071478 +0000 UTC m=+6.306696189 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-j6x28" (UniqueName: "kubernetes.io/projected/55619a2e-f6e9-4719-b362-30b69d553967-kube-api-access-j6x28") pod "network-check-target-w9tx8" (UID: "55619a2e-f6e9-4719-b362-30b69d553967") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:49:13.489622 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:13.488864    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cck7d"
Oct 30 21:49:13.489622 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:13.489003    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cck7d" podUID="b954bab2-9dd7-4b39-bda6-3fec87381aff"
Oct 30 21:49:13.489622 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:13.489105    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xbh9m"
Oct 30 21:49:13.489622 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:13.489200    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xbh9m" podUID="a6980f40-89cf-4f3d-8cd5-91969ae5a276"
Oct 30 21:49:13.489622 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:13.489267    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w9tx8"
Oct 30 21:49:13.489622 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:13.489335    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w9tx8" podUID="55619a2e-f6e9-4719-b362-30b69d553967"
Oct 30 21:49:13.536773 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:13.536727    2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 21:44:11 +0000 UTC" deadline="2028-04-26 04:36:04.358573268 +0000 UTC"
Oct 30 21:49:13.536773 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:13.536768    2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13038h46m50.821810236s"
Oct 30 21:49:15.102217 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:15.102149    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6980f40-89cf-4f3d-8cd5-91969ae5a276-metrics-certs\") pod \"network-metrics-daemon-xbh9m\" (UID: \"a6980f40-89cf-4f3d-8cd5-91969ae5a276\") " pod="openshift-multus/network-metrics-daemon-xbh9m"
Oct 30 21:49:15.102756 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:15.102273    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b954bab2-9dd7-4b39-bda6-3fec87381aff-original-pull-secret\") pod \"global-pull-secret-syncer-cck7d\" (UID: \"b954bab2-9dd7-4b39-bda6-3fec87381aff\") " pod="kube-system/global-pull-secret-syncer-cck7d"
Oct 30 21:49:15.102756 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:15.102451    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 21:49:15.102756 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:15.102521    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b954bab2-9dd7-4b39-bda6-3fec87381aff-original-pull-secret podName:b954bab2-9dd7-4b39-bda6-3fec87381aff nodeName:}" failed. No retries permitted until 2026-10-30 21:49:19.102500273 +0000 UTC m=+10.225124997 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b954bab2-9dd7-4b39-bda6-3fec87381aff-original-pull-secret") pod "global-pull-secret-syncer-cck7d" (UID: "b954bab2-9dd7-4b39-bda6-3fec87381aff") : object "kube-system"/"original-pull-secret" not registered
Oct 30 21:49:15.102756 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:15.102606    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:49:15.102756 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:15.102675    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6980f40-89cf-4f3d-8cd5-91969ae5a276-metrics-certs podName:a6980f40-89cf-4f3d-8cd5-91969ae5a276 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:19.102657794 +0000 UTC m=+10.225282508 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a6980f40-89cf-4f3d-8cd5-91969ae5a276-metrics-certs") pod "network-metrics-daemon-xbh9m" (UID: "a6980f40-89cf-4f3d-8cd5-91969ae5a276") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:49:15.203421 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:15.203322    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6x28\" (UniqueName: \"kubernetes.io/projected/55619a2e-f6e9-4719-b362-30b69d553967-kube-api-access-j6x28\") pod \"network-check-target-w9tx8\" (UID: \"55619a2e-f6e9-4719-b362-30b69d553967\") " pod="openshift-network-diagnostics/network-check-target-w9tx8"
Oct 30 21:49:15.203592 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:15.203545    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 21:49:15.203592 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:15.203569    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 21:49:15.203592 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:15.203584    2573 projected.go:196] Error preparing data for projected volume kube-api-access-j6x28 for pod openshift-network-diagnostics/network-check-target-w9tx8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:49:15.203750 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:15.203650    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/55619a2e-f6e9-4719-b362-30b69d553967-kube-api-access-j6x28 podName:55619a2e-f6e9-4719-b362-30b69d553967 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:19.203629254 +0000 UTC m=+10.326254022 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-j6x28" (UniqueName: "kubernetes.io/projected/55619a2e-f6e9-4719-b362-30b69d553967-kube-api-access-j6x28") pod "network-check-target-w9tx8" (UID: "55619a2e-f6e9-4719-b362-30b69d553967") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:49:15.483136 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:15.482538    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cck7d"
Oct 30 21:49:15.483136 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:15.482677    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cck7d" podUID="b954bab2-9dd7-4b39-bda6-3fec87381aff"
Oct 30 21:49:15.483136 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:15.482760    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xbh9m"
Oct 30 21:49:15.483136 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:15.482862    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xbh9m" podUID="a6980f40-89cf-4f3d-8cd5-91969ae5a276"
Oct 30 21:49:15.483136 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:15.482920    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w9tx8"
Oct 30 21:49:15.483136 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:15.482987    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w9tx8" podUID="55619a2e-f6e9-4719-b362-30b69d553967"
Oct 30 21:49:16.538170 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:16.537921    2573 generic.go:356] "Generic (PLEG): container finished" podID="d8412ee5d9c7152e66cb6933cbd6fbc8" containerID="72424879b5ac678cd40f9fd655a0a39b1bea72c1ff1d689e32996870e99d35ea" exitCode=0
Oct 30 21:49:16.538688 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:16.538108    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-245.us-west-2.compute.internal" event={"ID":"d8412ee5d9c7152e66cb6933cbd6fbc8","Type":"ContainerDied","Data":"72424879b5ac678cd40f9fd655a0a39b1bea72c1ff1d689e32996870e99d35ea"}
Oct 30 21:49:16.550618 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:16.550108    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-245.us-west-2.compute.internal" event={"ID":"bb28c8c054a6af006e24e10c141990f3","Type":"ContainerStarted","Data":"c531e24ed710fdc8c273ee2562bad7c804ee7684d3ed9abe8fbc89b9eb3227e6"}
Oct 30 21:49:16.565866 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:16.565147    2573 generic.go:356] "Generic (PLEG): container finished" podID="7a6305de-8fd1-4d3e-9628-8c4cda07f379" containerID="5e17da19f23daf9a0c51c74fd19a0094ab93c0ee4d6f7dbe83e2847177a17fad" exitCode=0
Oct 30 21:49:16.565866 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:16.565191    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vrtnk" event={"ID":"7a6305de-8fd1-4d3e-9628-8c4cda07f379","Type":"ContainerDied","Data":"5e17da19f23daf9a0c51c74fd19a0094ab93c0ee4d6f7dbe83e2847177a17fad"}
Oct 30 21:49:16.784039 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:16.783982    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-245.us-west-2.compute.internal" podStartSLOduration=5.783967523 podStartE2EDuration="5.783967523s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:49:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:16.783628724 +0000 UTC m=+7.906253458" watchObservedRunningTime="2026-10-30 21:49:16.783967523 +0000 UTC m=+7.906592256"
Oct 30 21:49:17.482961 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:17.482881    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w9tx8"
Oct 30 21:49:17.483129 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:17.483027    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w9tx8" podUID="55619a2e-f6e9-4719-b362-30b69d553967"
Oct 30 21:49:17.483193 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:17.483129    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cck7d"
Oct 30 21:49:17.483245 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:17.483216    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cck7d" podUID="b954bab2-9dd7-4b39-bda6-3fec87381aff"
Oct 30 21:49:17.483343 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:17.483302    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xbh9m"
Oct 30 21:49:17.483432 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:17.483406    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xbh9m" podUID="a6980f40-89cf-4f3d-8cd5-91969ae5a276"
Oct 30 21:49:17.571732 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:17.571692    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-245.us-west-2.compute.internal" event={"ID":"d8412ee5d9c7152e66cb6933cbd6fbc8","Type":"ContainerStarted","Data":"b6c09ab85e7691eed67138e66f9d1c5bd9f3bd17c0d1a4acfaf523cbb0a5f06a"}
Oct 30 21:49:17.578468 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:17.578437    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-w72c9" event={"ID":"0ad59239-edcc-46a4-a5a1-1debb63e3a5d","Type":"ContainerStarted","Data":"7894ed611ea9b236d5732a781a2814c4e738ee69758d6c4f73d4dd916f54f676"}
Oct 30 21:49:17.807025 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:17.806972    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-245.us-west-2.compute.internal" podStartSLOduration=6.8069513189999995 podStartE2EDuration="6.806951319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:49:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:17.688825099 +0000 UTC m=+8.811449828" watchObservedRunningTime="2026-10-30 21:49:17.806951319 +0000 UTC m=+8.929576049"
Oct 30 21:49:18.584051 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:18.584002    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-gmbbw" event={"ID":"82f75dd5-40cd-4cb3-9111-6213d6b43679","Type":"ContainerStarted","Data":"4b1c18a4dda982165d231a36949ab64aba8834cee9da69066be24871943fc063"}
Oct 30 21:49:18.694845 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:18.694786    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-w72c9" podStartSLOduration=3.467440852 podStartE2EDuration="8.694765865s" totalImagesPullingTime="5.227325013s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:49:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:17.807259293 +0000 UTC m=+8.929884021" watchObservedRunningTime="2026-10-30 21:49:18.694765865 +0000 UTC m=+9.817390596"
Oct 30 21:49:19.140880 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:19.140842    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b954bab2-9dd7-4b39-bda6-3fec87381aff-original-pull-secret\") pod \"global-pull-secret-syncer-cck7d\" (UID: \"b954bab2-9dd7-4b39-bda6-3fec87381aff\") " pod="kube-system/global-pull-secret-syncer-cck7d"
Oct 30 21:49:19.141055 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:19.140920    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6980f40-89cf-4f3d-8cd5-91969ae5a276-metrics-certs\") pod \"network-metrics-daemon-xbh9m\" (UID: \"a6980f40-89cf-4f3d-8cd5-91969ae5a276\") " pod="openshift-multus/network-metrics-daemon-xbh9m"
Oct 30 21:49:19.141120 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:19.141087    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:49:19.141173 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:19.141148    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6980f40-89cf-4f3d-8cd5-91969ae5a276-metrics-certs podName:a6980f40-89cf-4f3d-8cd5-91969ae5a276 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:27.141129703 +0000 UTC m=+18.263754413 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a6980f40-89cf-4f3d-8cd5-91969ae5a276-metrics-certs") pod "network-metrics-daemon-xbh9m" (UID: "a6980f40-89cf-4f3d-8cd5-91969ae5a276") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:49:19.141559 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:19.141545    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 21:49:19.141687 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:19.141599    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b954bab2-9dd7-4b39-bda6-3fec87381aff-original-pull-secret podName:b954bab2-9dd7-4b39-bda6-3fec87381aff nodeName:}" failed. No retries permitted until 2026-10-30 21:49:27.141583916 +0000 UTC m=+18.264208625 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b954bab2-9dd7-4b39-bda6-3fec87381aff-original-pull-secret") pod "global-pull-secret-syncer-cck7d" (UID: "b954bab2-9dd7-4b39-bda6-3fec87381aff") : object "kube-system"/"original-pull-secret" not registered
Oct 30 21:49:19.241871 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:19.241824    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6x28\" (UniqueName: \"kubernetes.io/projected/55619a2e-f6e9-4719-b362-30b69d553967-kube-api-access-j6x28\") pod \"network-check-target-w9tx8\" (UID: \"55619a2e-f6e9-4719-b362-30b69d553967\") " pod="openshift-network-diagnostics/network-check-target-w9tx8"
Oct 30 21:49:19.242030 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:19.241995    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 21:49:19.242030 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:19.242017    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 21:49:19.242030 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:19.242027    2573 projected.go:196] Error preparing data for projected volume kube-api-access-j6x28 for pod openshift-network-diagnostics/network-check-target-w9tx8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:49:19.242180 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:19.242111    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/55619a2e-f6e9-4719-b362-30b69d553967-kube-api-access-j6x28 podName:55619a2e-f6e9-4719-b362-30b69d553967 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:27.242094129 +0000 UTC m=+18.364718858 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-j6x28" (UniqueName: "kubernetes.io/projected/55619a2e-f6e9-4719-b362-30b69d553967-kube-api-access-j6x28") pod "network-check-target-w9tx8" (UID: "55619a2e-f6e9-4719-b362-30b69d553967") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:49:19.483532 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:19.483446    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xbh9m"
Oct 30 21:49:19.483679 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:19.483562    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xbh9m" podUID="a6980f40-89cf-4f3d-8cd5-91969ae5a276"
Oct 30 21:49:19.483679 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:19.483625    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w9tx8"
Oct 30 21:49:19.483802 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:19.483688    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w9tx8" podUID="55619a2e-f6e9-4719-b362-30b69d553967"
Oct 30 21:49:19.483802 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:19.483738    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cck7d"
Oct 30 21:49:19.483921 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:19.483810    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cck7d" podUID="b954bab2-9dd7-4b39-bda6-3fec87381aff"
Oct 30 21:49:19.590116 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:19.590076    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lrc2c" event={"ID":"924394f7-5b3b-4767-a1ff-f1d310ece4ed","Type":"ContainerStarted","Data":"7bbab7ce441b36c281eca490e5343d7b120e2af5761f317b39d9b3392af698ab"}
Oct 30 21:49:20.121792 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:20.121752    2573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-w72c9"
Oct 30 21:49:20.122735 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:20.122701    2573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-w72c9"
Oct 30 21:49:20.233233 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:20.233179    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-gmbbw" podStartSLOduration=3.949300586 podStartE2EDuration="10.233157878s" totalImagesPullingTime="6.283857292s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:49:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:18.695323622 +0000 UTC m=+9.817948355" watchObservedRunningTime="2026-10-30 21:49:20.233157878 +0000 UTC m=+11.355782609"
Oct 30 21:49:20.596931 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:20.596846    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-p7zpx" event={"ID":"5f468f3c-bf3d-437c-8ec8-a5d56f448d99","Type":"ContainerStarted","Data":"8c70c15f69ad58f6581ee88f6714d344e3ecde33879d9cee03ed3e87a41c6dce"}
Oct 30 21:49:20.596931 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:20.596905    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-w72c9"
Oct 30 21:49:20.598027 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:20.598008    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-w72c9"
Oct 30 21:49:20.716981 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:20.716776    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-p7zpx" podStartSLOduration=2.305460497 podStartE2EDuration="10.716757477s" totalImagesPullingTime="8.41129698s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:49:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:20.715930822 +0000 UTC m=+11.838555533" watchObservedRunningTime="2026-10-30 21:49:20.716757477 +0000 UTC m=+11.839382206"
Oct 30 21:49:20.861693 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:20.861665    2573 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 21:49:21.426670 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:21.426561    2573 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T21:49:20.861690857Z","UUID":"2429a1b8-5cc5-4fa3-937c-522de5e03443","Handler":null,"Name":"","Endpoint":""}
Oct 30 21:49:21.428372 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:21.428340    2573 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 21:49:21.428372 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:21.428374    2573 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 21:49:21.479976 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:21.479895    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xbh9m"
Oct 30 21:49:21.480107 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:21.479982    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cck7d"
Oct 30 21:49:21.480107 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:21.479995    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w9tx8"
Oct 30 21:49:21.480190 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:21.480102    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xbh9m" podUID="a6980f40-89cf-4f3d-8cd5-91969ae5a276"
Oct 30 21:49:21.480272 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:21.480251    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cck7d" podUID="b954bab2-9dd7-4b39-bda6-3fec87381aff"
Oct 30 21:49:21.480358 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:21.480341    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w9tx8" podUID="55619a2e-f6e9-4719-b362-30b69d553967"
Oct 30 21:49:21.600680 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:21.600642    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gbpvx" event={"ID":"32d63b18-0fcb-4e4a-a24c-ee044dcb5a90","Type":"ContainerStarted","Data":"22f54c3c0954e19e954816b17769c819773f89c191c5d46a27c4e2e8ec71a2b5"}
Oct 30 21:49:21.603211 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:21.603185    2573 generic.go:356] "Generic (PLEG): container finished" podID="7a6305de-8fd1-4d3e-9628-8c4cda07f379" containerID="f349a356942ed99fcc929e902dc50e6cee3c4a40e4efe6d1f46a6d0a549d61a7" exitCode=0
Oct 30 21:49:21.603315 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:21.603266    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vrtnk" event={"ID":"7a6305de-8fd1-4d3e-9628-8c4cda07f379","Type":"ContainerDied","Data":"f349a356942ed99fcc929e902dc50e6cee3c4a40e4efe6d1f46a6d0a549d61a7"}
Oct 30 21:49:21.606423 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:21.606294    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lrc2c" event={"ID":"924394f7-5b3b-4767-a1ff-f1d310ece4ed","Type":"ContainerStarted","Data":"6aa01d1b90ec530a3ebb2681d5edc05eefde7d8a3edcc4a2097464027c2dfedd"}
Oct 30 21:49:21.711452 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:21.711380    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-gbpvx" podStartSLOduration=2.829216385 podStartE2EDuration="11.711366896s" totalImagesPullingTime="8.882150511s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:49:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:21.710835208 +0000 UTC m=+12.833459941" watchObservedRunningTime="2026-10-30 21:49:21.711366896 +0000 UTC m=+12.833991624"
Oct 30 21:49:22.610478 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:22.610434    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-t24lz" event={"ID":"f22e4514-ac6d-481d-8afe-6e2e4cdc3aa1","Type":"ContainerStarted","Data":"37f808671ae853ddfee14419a641d210a3a03f61ffad462e64ad774f9b41bf83"}
Oct 30 21:49:22.720169 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:22.720122    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-t24lz" podStartSLOduration=3.47582677 podStartE2EDuration="12.720109961s" totalImagesPullingTime="9.244283191s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:49:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:22.719888737 +0000 UTC m=+13.842513464" watchObservedRunningTime="2026-10-30 21:49:22.720109961 +0000 UTC m=+13.842734688"
Oct 30 21:49:23.479974 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:23.479871    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xbh9m"
Oct 30 21:49:23.480090 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:23.480026    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xbh9m" podUID="a6980f40-89cf-4f3d-8cd5-91969ae5a276"
Oct 30 21:49:23.480090 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:23.480069    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w9tx8"
Oct 30 21:49:23.480197 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:23.480151    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w9tx8" podUID="55619a2e-f6e9-4719-b362-30b69d553967"
Oct 30 21:49:23.480197 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:23.480195    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cck7d"
Oct 30 21:49:23.480294 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:23.480244    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cck7d" podUID="b954bab2-9dd7-4b39-bda6-3fec87381aff"
Oct 30 21:49:23.614796 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:23.614759    2573 generic.go:356] "Generic (PLEG): container finished" podID="7a6305de-8fd1-4d3e-9628-8c4cda07f379" containerID="9d4a8af82f0954e2903c897898b8de663da1eca550d940952fe5f1253cb6d2b2" exitCode=0
Oct 30 21:49:23.615584 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:23.614841    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vrtnk" event={"ID":"7a6305de-8fd1-4d3e-9628-8c4cda07f379","Type":"ContainerDied","Data":"9d4a8af82f0954e2903c897898b8de663da1eca550d940952fe5f1253cb6d2b2"}
Oct 30 21:49:23.617518 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:23.617496    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lrc2c" event={"ID":"924394f7-5b3b-4767-a1ff-f1d310ece4ed","Type":"ContainerStarted","Data":"5a6f45a09b56370e67620136c520b2103396e872c96619ff5ea099eff9c494f1"}
Oct 30 21:49:23.619107 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:23.619084    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-22pl2" event={"ID":"d947d3db-ab6e-4874-8817-f8c74be138a1","Type":"ContainerStarted","Data":"9fb69f5f37b0690460e9f17c415f1a62f8e0bef14bde6f1f49d77d8cc241cc54"}
Oct 30 21:49:23.883324 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:23.883209    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-22pl2" podStartSLOduration=2.520057369 podStartE2EDuration="13.883192523s" totalImagesPullingTime="11.363135154s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:49:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:23.88315655 +0000 UTC m=+15.005781278" watchObservedRunningTime="2026-10-30 21:49:23.883192523 +0000 UTC m=+15.005817252"
Oct 30 21:49:23.997887 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:23.997835    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lrc2c" podStartSLOduration=3.332817264 podStartE2EDuration="13.997816573s" totalImagesPullingTime="10.664999309s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:49:10 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:23.997116432 +0000 UTC m=+15.119741161" watchObservedRunningTime="2026-10-30 21:49:23.997816573 +0000 UTC m=+15.120441301"
Oct 30 21:49:25.480067 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:25.479883    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xbh9m"
Oct 30 21:49:25.480640 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:25.479874    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w9tx8"
Oct 30 21:49:25.480640 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:25.480177    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xbh9m" podUID="a6980f40-89cf-4f3d-8cd5-91969ae5a276"
Oct 30 21:49:25.480640 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:25.479908    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cck7d"
Oct 30 21:49:25.480640 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:25.480236    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w9tx8" podUID="55619a2e-f6e9-4719-b362-30b69d553967"
Oct 30 21:49:25.480640 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:25.480316    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cck7d" podUID="b954bab2-9dd7-4b39-bda6-3fec87381aff"
Oct 30 21:49:25.626255 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:25.626218    2573 generic.go:356] "Generic (PLEG): container finished" podID="7a6305de-8fd1-4d3e-9628-8c4cda07f379" containerID="82a62e13298910f03ade52d27ed8119eadace52fd94746cb80d6dc8beddbabd0" exitCode=0
Oct 30 21:49:25.626427 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:25.626268    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vrtnk" event={"ID":"7a6305de-8fd1-4d3e-9628-8c4cda07f379","Type":"ContainerDied","Data":"82a62e13298910f03ade52d27ed8119eadace52fd94746cb80d6dc8beddbabd0"}
Oct 30 21:49:27.206133 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:27.206101    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6980f40-89cf-4f3d-8cd5-91969ae5a276-metrics-certs\") pod \"network-metrics-daemon-xbh9m\" (UID: \"a6980f40-89cf-4f3d-8cd5-91969ae5a276\") " pod="openshift-multus/network-metrics-daemon-xbh9m"
Oct 30 21:49:27.206544 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:27.206178    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b954bab2-9dd7-4b39-bda6-3fec87381aff-original-pull-secret\") pod \"global-pull-secret-syncer-cck7d\" (UID: \"b954bab2-9dd7-4b39-bda6-3fec87381aff\") " pod="kube-system/global-pull-secret-syncer-cck7d"
Oct 30 21:49:27.206544 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:27.206255    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:49:27.206544 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:27.206269    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 21:49:27.206544 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:27.206320    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b954bab2-9dd7-4b39-bda6-3fec87381aff-original-pull-secret podName:b954bab2-9dd7-4b39-bda6-3fec87381aff nodeName:}" failed. No retries permitted until 2026-10-30 21:49:43.206306248 +0000 UTC m=+34.328930967 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b954bab2-9dd7-4b39-bda6-3fec87381aff-original-pull-secret") pod "global-pull-secret-syncer-cck7d" (UID: "b954bab2-9dd7-4b39-bda6-3fec87381aff") : object "kube-system"/"original-pull-secret" not registered
Oct 30 21:49:27.206544 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:27.206333    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6980f40-89cf-4f3d-8cd5-91969ae5a276-metrics-certs podName:a6980f40-89cf-4f3d-8cd5-91969ae5a276 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:43.20632678 +0000 UTC m=+34.328951499 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a6980f40-89cf-4f3d-8cd5-91969ae5a276-metrics-certs") pod "network-metrics-daemon-xbh9m" (UID: "a6980f40-89cf-4f3d-8cd5-91969ae5a276") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:49:27.306874 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:27.306843    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6x28\" (UniqueName: \"kubernetes.io/projected/55619a2e-f6e9-4719-b362-30b69d553967-kube-api-access-j6x28\") pod \"network-check-target-w9tx8\" (UID: \"55619a2e-f6e9-4719-b362-30b69d553967\") " pod="openshift-network-diagnostics/network-check-target-w9tx8"
Oct 30 21:49:27.307036 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:27.307016    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 21:49:27.307036 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:27.307034    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 21:49:27.307129 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:27.307046    2573 projected.go:196] Error preparing data for projected volume kube-api-access-j6x28 for pod openshift-network-diagnostics/network-check-target-w9tx8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:49:27.307129 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:27.307092    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/55619a2e-f6e9-4719-b362-30b69d553967-kube-api-access-j6x28 podName:55619a2e-f6e9-4719-b362-30b69d553967 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:43.307077063 +0000 UTC m=+34.429701769 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-j6x28" (UniqueName: "kubernetes.io/projected/55619a2e-f6e9-4719-b362-30b69d553967-kube-api-access-j6x28") pod "network-check-target-w9tx8" (UID: "55619a2e-f6e9-4719-b362-30b69d553967") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:49:27.480641 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:27.480443    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xbh9m"
Oct 30 21:49:27.480754 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:27.480495    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cck7d"
Oct 30 21:49:27.480813 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:27.480752    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xbh9m" podUID="a6980f40-89cf-4f3d-8cd5-91969ae5a276"
Oct 30 21:49:27.480813 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:27.480539    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w9tx8"
Oct 30 21:49:27.480908 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:27.480823    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cck7d" podUID="b954bab2-9dd7-4b39-bda6-3fec87381aff"
Oct 30 21:49:27.480908 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:27.480880    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w9tx8" podUID="55619a2e-f6e9-4719-b362-30b69d553967"
Oct 30 21:49:27.633390 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:27.633357    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf" event={"ID":"41fe6d63-9316-4916-b042-87ba7e602167","Type":"ContainerStarted","Data":"63b3e60b856f421b626c0c396acfdff113c1216153dde4dc75558b67d2cac909"}
Oct 30 21:49:27.633548 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:27.633457    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf" event={"ID":"41fe6d63-9316-4916-b042-87ba7e602167","Type":"ContainerStarted","Data":"5544abb964fba21452909ca0e413c208767d94ff8afd456e60ffb218940e371e"}
Oct 30 21:49:27.633548 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:27.633494    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf" event={"ID":"41fe6d63-9316-4916-b042-87ba7e602167","Type":"ContainerStarted","Data":"611587bb3db4aef07036acfc199c7979d4eb0ca1674261037661ec740e919773"}
Oct 30 21:49:27.633548 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:27.633521    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf" event={"ID":"41fe6d63-9316-4916-b042-87ba7e602167","Type":"ContainerStarted","Data":"b1024154b4e0a5fe6b673dc47165b4349dbb33b716c25536cb9fda6f30c99fcf"}
Oct 30 21:49:27.633681 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:27.633563    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf" event={"ID":"41fe6d63-9316-4916-b042-87ba7e602167","Type":"ContainerStarted","Data":"c66c805269ad61adae61f4bc2c551b0e83cf6b40ed7c5a19c52de74fea049b71"}
Oct 30 21:49:27.633681 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:27.633609    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf" event={"ID":"41fe6d63-9316-4916-b042-87ba7e602167","Type":"ContainerStarted","Data":"def8b0f9fd0e30cd0eafdde189ae62e84fa51c173235e5d5468734078ec02a17"}
Oct 30 21:49:28.638527 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:28.638491    2573 generic.go:356] "Generic (PLEG): container finished" podID="7a6305de-8fd1-4d3e-9628-8c4cda07f379" containerID="3c1c44d1979b97f91d92a71a5f56aed61bf71890981234dc6b289cb0763da087" exitCode=0
Oct 30 21:49:28.639166 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:28.638549    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vrtnk" event={"ID":"7a6305de-8fd1-4d3e-9628-8c4cda07f379","Type":"ContainerDied","Data":"3c1c44d1979b97f91d92a71a5f56aed61bf71890981234dc6b289cb0763da087"}
Oct 30 21:49:29.310640 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:29.310615    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-gbpvx_32d63b18-0fcb-4e4a-a24c-ee044dcb5a90/dns-node-resolver/0.log"
Oct 30 21:49:29.483142 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:29.483114    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w9tx8"
Oct 30 21:49:29.483291 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:29.483188    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xbh9m"
Oct 30 21:49:29.483291 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:29.483213    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cck7d"
Oct 30 21:49:29.483422 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:29.483322    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xbh9m" podUID="a6980f40-89cf-4f3d-8cd5-91969ae5a276"
Oct 30 21:49:29.483422 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:29.483353    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cck7d" podUID="b954bab2-9dd7-4b39-bda6-3fec87381aff"
Oct 30 21:49:29.483525 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:29.483428    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w9tx8" podUID="55619a2e-f6e9-4719-b362-30b69d553967"
Oct 30 21:49:29.643837 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:29.643808    2573 generic.go:356] "Generic (PLEG): container finished" podID="7a6305de-8fd1-4d3e-9628-8c4cda07f379" containerID="2a80cb2c3c33e04cec680df1a093e34cab4c2a5ae1862fad0dff04b2e211caef" exitCode=0
Oct 30 21:49:29.644377 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:29.643895    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vrtnk" event={"ID":"7a6305de-8fd1-4d3e-9628-8c4cda07f379","Type":"ContainerDied","Data":"2a80cb2c3c33e04cec680df1a093e34cab4c2a5ae1862fad0dff04b2e211caef"}
Oct 30 21:49:29.647449 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:29.647384    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf" event={"ID":"41fe6d63-9316-4916-b042-87ba7e602167","Type":"ContainerStarted","Data":"36658d9140ba469937db94d183b2bbbc0429020f005b0265706202aa70ddaf0d"}
Oct 30 21:49:29.914190 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:29.914170    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-gmbbw_82f75dd5-40cd-4cb3-9111-6213d6b43679/node-ca/0.log"
Oct 30 21:49:30.652950 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:30.652911    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vrtnk" event={"ID":"7a6305de-8fd1-4d3e-9628-8c4cda07f379","Type":"ContainerStarted","Data":"e63c4fbae344e8ead706f77ac64ce2abbeb4241e2981b143d8e11e44289a5051"}
Oct 30 21:49:30.770921 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:30.770878    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-vrtnk" podStartSLOduration=7.536700661 podStartE2EDuration="20.770864639s" totalImagesPullingTime="13.234163978s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:49:10 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:30.770793739 +0000 UTC m=+21.893418467" watchObservedRunningTime="2026-10-30 21:49:30.770864639 +0000 UTC m=+21.893489367"
Oct 30 21:49:31.480370 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:31.480336    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xbh9m"
Oct 30 21:49:31.480370 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:31.480363    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cck7d"
Oct 30 21:49:31.480630 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:31.480336    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w9tx8"
Oct 30 21:49:31.480630 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:31.480480    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xbh9m" podUID="a6980f40-89cf-4f3d-8cd5-91969ae5a276"
Oct 30 21:49:31.480630 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:31.480565    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cck7d" podUID="b954bab2-9dd7-4b39-bda6-3fec87381aff"
Oct 30 21:49:31.480783 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:31.480633    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w9tx8" podUID="55619a2e-f6e9-4719-b362-30b69d553967"
Oct 30 21:49:32.659895 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:32.659652    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf" event={"ID":"41fe6d63-9316-4916-b042-87ba7e602167","Type":"ContainerStarted","Data":"95e2d50927a7c1b5976f3f7b7c55f005b2029c1f9a15b780adc8f8bd34a24ad4"}
Oct 30 21:49:32.660420 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:32.660038    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf"
Oct 30 21:49:32.660420 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:32.660092    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf"
Oct 30 21:49:32.674742 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:32.674719    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf"
Oct 30 21:49:32.674847 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:32.674824    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf"
Oct 30 21:49:32.923996 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:32.923895    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf" podStartSLOduration=7.512983409 podStartE2EDuration="22.923881963s" totalImagesPullingTime="15.410898554s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:49:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:32.923852575 +0000 UTC m=+24.046477314" watchObservedRunningTime="2026-10-30 21:49:32.923881963 +0000 UTC m=+24.046506691"
Oct 30 21:49:33.479837 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:33.479802    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w9tx8"
Oct 30 21:49:33.479837 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:33.479827    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xbh9m"
Oct 30 21:49:33.480055 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:33.479912    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w9tx8" podUID="55619a2e-f6e9-4719-b362-30b69d553967"
Oct 30 21:49:33.480055 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:33.479955    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cck7d"
Oct 30 21:49:33.480055 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:33.480026    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cck7d" podUID="b954bab2-9dd7-4b39-bda6-3fec87381aff"
Oct 30 21:49:33.480227 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:33.480098    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xbh9m" podUID="a6980f40-89cf-4f3d-8cd5-91969ae5a276"
Oct 30 21:49:33.661130 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:33.661103    2573 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 21:49:34.664262 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:34.664236    2573 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 21:49:35.480142 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:35.480110    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cck7d"
Oct 30 21:49:35.480306 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:35.480106    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w9tx8"
Oct 30 21:49:35.480306 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:35.480232    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cck7d" podUID="b954bab2-9dd7-4b39-bda6-3fec87381aff"
Oct 30 21:49:35.480306 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:35.480124    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xbh9m"
Oct 30 21:49:35.480306 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:35.480275    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w9tx8" podUID="55619a2e-f6e9-4719-b362-30b69d553967"
Oct 30 21:49:35.480542 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:35.480346    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xbh9m" podUID="a6980f40-89cf-4f3d-8cd5-91969ae5a276"
Oct 30 21:49:36.568626 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:36.568594    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf"
Oct 30 21:49:36.569089 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:36.568828    2573 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 21:49:36.578320 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:36.578284    2573 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf" podUID="41fe6d63-9316-4916-b042-87ba7e602167" containerName="ovnkube-controller" probeResult="failure" output=""
Oct 30 21:49:36.586499 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:36.586467    2573 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf" podUID="41fe6d63-9316-4916-b042-87ba7e602167" containerName="ovnkube-controller" probeResult="failure" output=""
Oct 30 21:49:37.480132 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:37.480103    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w9tx8"
Oct 30 21:49:37.480132 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:37.480129    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xbh9m"
Oct 30 21:49:37.480503 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:37.480204    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w9tx8" podUID="55619a2e-f6e9-4719-b362-30b69d553967"
Oct 30 21:49:37.480503 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:37.480256    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cck7d"
Oct 30 21:49:37.480503 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:37.480357    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cck7d" podUID="b954bab2-9dd7-4b39-bda6-3fec87381aff"
Oct 30 21:49:37.480503 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:37.480460    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xbh9m" podUID="a6980f40-89cf-4f3d-8cd5-91969ae5a276"
Oct 30 21:49:39.485575 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:39.485547    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xbh9m"
Oct 30 21:49:39.485575 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:39.485564    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cck7d"
Oct 30 21:49:39.485575 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:39.485555    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w9tx8"
Oct 30 21:49:39.485991 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:39.485702    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cck7d" podUID="b954bab2-9dd7-4b39-bda6-3fec87381aff"
Oct 30 21:49:39.485991 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:39.485718    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w9tx8" podUID="55619a2e-f6e9-4719-b362-30b69d553967"
Oct 30 21:49:39.485991 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:39.485794    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xbh9m" podUID="a6980f40-89cf-4f3d-8cd5-91969ae5a276"
Oct 30 21:49:41.480418 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:41.480382    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cck7d"
Oct 30 21:49:41.480418 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:41.480389    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xbh9m"
Oct 30 21:49:41.480861 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:41.480500    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cck7d" podUID="b954bab2-9dd7-4b39-bda6-3fec87381aff"
Oct 30 21:49:41.480861 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:41.480577    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xbh9m" podUID="a6980f40-89cf-4f3d-8cd5-91969ae5a276"
Oct 30 21:49:41.480861 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:41.480605    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w9tx8"
Oct 30 21:49:41.480861 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:41.480700    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w9tx8" podUID="55619a2e-f6e9-4719-b362-30b69d553967"
Oct 30 21:49:43.230695 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:43.230664    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b954bab2-9dd7-4b39-bda6-3fec87381aff-original-pull-secret\") pod \"global-pull-secret-syncer-cck7d\" (UID: \"b954bab2-9dd7-4b39-bda6-3fec87381aff\") " pod="kube-system/global-pull-secret-syncer-cck7d"
Oct 30 21:49:43.231071 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:43.230711    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6980f40-89cf-4f3d-8cd5-91969ae5a276-metrics-certs\") pod \"network-metrics-daemon-xbh9m\" (UID: \"a6980f40-89cf-4f3d-8cd5-91969ae5a276\") " pod="openshift-multus/network-metrics-daemon-xbh9m"
Oct 30 21:49:43.231071 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:43.230815    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:49:43.231071 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:43.230870    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6980f40-89cf-4f3d-8cd5-91969ae5a276-metrics-certs podName:a6980f40-89cf-4f3d-8cd5-91969ae5a276 nodeName:}" failed. No retries permitted until 2026-10-30 21:50:15.230857668 +0000 UTC m=+66.353482373 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a6980f40-89cf-4f3d-8cd5-91969ae5a276-metrics-certs") pod "network-metrics-daemon-xbh9m" (UID: "a6980f40-89cf-4f3d-8cd5-91969ae5a276") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:49:43.231071 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:43.230815    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 21:49:43.231071 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:43.230943    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b954bab2-9dd7-4b39-bda6-3fec87381aff-original-pull-secret podName:b954bab2-9dd7-4b39-bda6-3fec87381aff nodeName:}" failed. No retries permitted until 2026-10-30 21:50:15.23093042 +0000 UTC m=+66.353555142 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b954bab2-9dd7-4b39-bda6-3fec87381aff-original-pull-secret") pod "global-pull-secret-syncer-cck7d" (UID: "b954bab2-9dd7-4b39-bda6-3fec87381aff") : object "kube-system"/"original-pull-secret" not registered
Oct 30 21:49:43.331538 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:43.331505    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6x28\" (UniqueName: \"kubernetes.io/projected/55619a2e-f6e9-4719-b362-30b69d553967-kube-api-access-j6x28\") pod \"network-check-target-w9tx8\" (UID: \"55619a2e-f6e9-4719-b362-30b69d553967\") " pod="openshift-network-diagnostics/network-check-target-w9tx8"
Oct 30 21:49:43.331686 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:43.331637    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 21:49:43.331686 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:43.331653    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 21:49:43.331686 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:43.331664    2573 projected.go:196] Error preparing data for projected volume kube-api-access-j6x28 for pod openshift-network-diagnostics/network-check-target-w9tx8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:49:43.331797 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:43.331712    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/55619a2e-f6e9-4719-b362-30b69d553967-kube-api-access-j6x28 podName:55619a2e-f6e9-4719-b362-30b69d553967 nodeName:}" failed. No retries permitted until 2026-10-30 21:50:15.331701264 +0000 UTC m=+66.454325970 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-j6x28" (UniqueName: "kubernetes.io/projected/55619a2e-f6e9-4719-b362-30b69d553967-kube-api-access-j6x28") pod "network-check-target-w9tx8" (UID: "55619a2e-f6e9-4719-b362-30b69d553967") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:49:43.480270 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:43.480237    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cck7d"
Oct 30 21:49:43.480441 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:43.480239    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xbh9m"
Oct 30 21:49:43.480441 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:43.480341    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cck7d" podUID="b954bab2-9dd7-4b39-bda6-3fec87381aff"
Oct 30 21:49:43.480441 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:43.480239    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w9tx8"
Oct 30 21:49:43.480559 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:43.480448    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xbh9m" podUID="a6980f40-89cf-4f3d-8cd5-91969ae5a276"
Oct 30 21:49:43.480559 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:43.480482    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w9tx8" podUID="55619a2e-f6e9-4719-b362-30b69d553967"
Oct 30 21:49:45.480358 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:45.480326    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w9tx8"
Oct 30 21:49:45.480864 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:45.480345    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cck7d"
Oct 30 21:49:45.480864 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:45.480500    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xbh9m"
Oct 30 21:49:45.480864 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:45.480513    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w9tx8" podUID="55619a2e-f6e9-4719-b362-30b69d553967"
Oct 30 21:49:45.480864 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:45.480588    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cck7d" podUID="b954bab2-9dd7-4b39-bda6-3fec87381aff"
Oct 30 21:49:45.480864 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:45.480661    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xbh9m" podUID="a6980f40-89cf-4f3d-8cd5-91969ae5a276"
Oct 30 21:49:47.480197 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:47.480167    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w9tx8"
Oct 30 21:49:47.480197 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:47.480179    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cck7d"
Oct 30 21:49:47.480718 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:47.480186    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xbh9m"
Oct 30 21:49:47.480718 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:47.480257    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w9tx8" podUID="55619a2e-f6e9-4719-b362-30b69d553967"
Oct 30 21:49:47.480718 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:47.480430    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cck7d" podUID="b954bab2-9dd7-4b39-bda6-3fec87381aff"
Oct 30 21:49:47.480718 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:47.480506    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xbh9m" podUID="a6980f40-89cf-4f3d-8cd5-91969ae5a276"
Oct 30 21:49:49.481141 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:49.481111    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xbh9m"
Oct 30 21:49:49.481567 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:49.481218    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w9tx8"
Oct 30 21:49:49.481567 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:49.481262    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cck7d"
Oct 30 21:49:49.481567 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:49.481256    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xbh9m" podUID="a6980f40-89cf-4f3d-8cd5-91969ae5a276"
Oct 30 21:49:49.481567 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:49.481351    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cck7d" podUID="b954bab2-9dd7-4b39-bda6-3fec87381aff"
Oct 30 21:49:49.481567 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:49.481452    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w9tx8" podUID="55619a2e-f6e9-4719-b362-30b69d553967"
Oct 30 21:49:50.424143 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:50.423919    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-w9tx8"]
Oct 30 21:49:50.424275 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:50.424240    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w9tx8"
Oct 30 21:49:50.424789 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:50.424346    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w9tx8" podUID="55619a2e-f6e9-4719-b362-30b69d553967"
Oct 30 21:49:50.426023 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:50.426001    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-cck7d"]
Oct 30 21:49:50.426128 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:50.426107    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cck7d"
Oct 30 21:49:50.426243 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:50.426202    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cck7d" podUID="b954bab2-9dd7-4b39-bda6-3fec87381aff"
Oct 30 21:49:50.426922 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:50.426898    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xbh9m"]
Oct 30 21:49:50.427028 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:50.426996    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xbh9m"
Oct 30 21:49:50.427138 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:50.427117    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xbh9m" podUID="a6980f40-89cf-4f3d-8cd5-91969ae5a276"
Oct 30 21:49:51.480894 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:51.480868    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xbh9m"
Oct 30 21:49:51.481201 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:51.481001    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xbh9m" podUID="a6980f40-89cf-4f3d-8cd5-91969ae5a276"
Oct 30 21:49:52.480474 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:52.480447    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w9tx8"
Oct 30 21:49:52.480625 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:52.480541    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w9tx8" podUID="55619a2e-f6e9-4719-b362-30b69d553967"
Oct 30 21:49:52.480625 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:52.480594    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cck7d"
Oct 30 21:49:52.480697 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:52.480682    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cck7d" podUID="b954bab2-9dd7-4b39-bda6-3fec87381aff"
Oct 30 21:49:53.480815 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:53.480748    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xbh9m"
Oct 30 21:49:53.481223 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:53.480858    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xbh9m" podUID="a6980f40-89cf-4f3d-8cd5-91969ae5a276"
Oct 30 21:49:54.480452 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:54.480419    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w9tx8"
Oct 30 21:49:54.480606 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:54.480543    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w9tx8" podUID="55619a2e-f6e9-4719-b362-30b69d553967"
Oct 30 21:49:54.480606 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:54.480590    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cck7d"
Oct 30 21:49:54.480726 ip-10-0-128-245 kubenswrapper[2573]: E1030 21:49:54.480678    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cck7d" podUID="b954bab2-9dd7-4b39-bda6-3fec87381aff"
Oct 30 21:49:55.065726 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:55.065699    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-245.us-west-2.compute.internal" event="NodeReady"
Oct 30 21:49:55.066153 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:55.065826    2573 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 30 21:49:55.178626 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:55.178565    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-hr227"]
Oct 30 21:49:55.182370 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:55.182344    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hr227"
Oct 30 21:49:55.213978 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:55.213952    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66wt8\" (UniqueName: \"kubernetes.io/projected/516573f9-4ada-4868-b885-f61b9640e6bf-kube-api-access-66wt8\") pod \"ingress-canary-hr227\" (UID: \"516573f9-4ada-4868-b885-f61b9640e6bf\") " pod="openshift-ingress-canary/ingress-canary-hr227"
Oct 30 21:49:55.214066 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:55.213982    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/516573f9-4ada-4868-b885-f61b9640e6bf-cert\") pod \"ingress-canary-hr227\" (UID: \"516573f9-4ada-4868-b885-f61b9640e6bf\") " pod="openshift-ingress-canary/ingress-canary-hr227"
Oct 30 21:49:55.225365 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:55.225345    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-kn6xx"]
Oct 30 21:49:55.227253 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:55.227239    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-fdmdz"]
Oct 30 21:49:55.227361 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:55.227346    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-kn6xx"
Oct 30 21:49:55.229043 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:55.229024    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-69874d4cdd-zhlp5"]
Oct 30 21:49:55.229143 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:55.229101    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-fdmdz"
Oct 30 21:49:55.231536 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:55.231519    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hr227"]
Oct 30 21:49:55.231623 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:55.231600    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69874d4cdd-zhlp5"
Oct 30 21:49:55.233577 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:55.233562    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 21:49:55.236511 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:55.236494    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 21:49:55.240859 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:55.240836    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-kn6xx"]
Oct 30 21:49:55.272206 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:55.272181    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-69874d4cdd-zhlp5"]
Oct 30 21:49:55.300801 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:55.300772    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-fdmdz"]
Oct 30 21:49:55.315056 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:55.315032    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/87c44883-54d0-4a05-b94b-d3c2835a03e1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kn6xx\" (UID: \"87c44883-54d0-4a05-b94b-d3c2835a03e1\") " pod="openshift-insights/insights-runtime-extractor-kn6xx"
Oct 30 21:49:55.315169 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:55.315070    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/49ff8cb6-54de-4e8c-9a35-634f9a1ef29b-config-volume\") pod \"dns-default-fdmdz\" (UID: \"49ff8cb6-54de-4e8c-9a35-634f9a1ef29b\") " pod="openshift-dns/dns-default-fdmdz"
Oct 30 21:49:55.315169 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:55.315097    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/69e218ec-ab5e-4e09-8444-e53a710c53bf-console-serving-cert\") pod \"console-69874d4cdd-zhlp5\" (UID: \"69e218ec-ab5e-4e09-8444-e53a710c53bf\") " pod="openshift-console/console-69874d4cdd-zhlp5"
Oct 30 21:49:55.315249 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:55.315172    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49ff8cb6-54de-4e8c-9a35-634f9a1ef29b-tmp-dir\") pod \"dns-default-fdmdz\" (UID: \"49ff8cb6-54de-4e8c-9a35-634f9a1ef29b\") " pod="openshift-dns/dns-default-fdmdz"
Oct 30 21:49:55.315249 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:55.315218    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gv6wf\" (UniqueName: \"kubernetes.io/projected/69e218ec-ab5e-4e09-8444-e53a710c53bf-kube-api-access-gv6wf\") pod \"console-69874d4cdd-zhlp5\" (UID: \"69e218ec-ab5e-4e09-8444-e53a710c53bf\") " pod="openshift-console/console-69874d4cdd-zhlp5"
Oct 30 21:49:55.315320 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:55.315247    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7qxb\" (UniqueName: \"kubernetes.io/projected/49ff8cb6-54de-4e8c-9a35-634f9a1ef29b-kube-api-access-z7qxb\") pod \"dns-default-fdmdz\" (UID: \"49ff8cb6-54de-4e8c-9a35-634f9a1ef29b\") " pod="openshift-dns/dns-default-fdmdz"
Oct 30 21:49:55.315320 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:55.315265    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/69e218ec-ab5e-4e09-8444-e53a710c53bf-console-config\") pod \"console-69874d4cdd-zhlp5\" (UID: \"69e218ec-ab5e-4e09-8444-e53a710c53bf\") " pod="openshift-console/console-69874d4cdd-zhlp5"
Oct 30 21:49:55.315320 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:55.315287    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/87c44883-54d0-4a05-b94b-d3c2835a03e1-crio-socket\") pod \"insights-runtime-extractor-kn6xx\" (UID: \"87c44883-54d0-4a05-b94b-d3c2835a03e1\") " pod="openshift-insights/insights-runtime-extractor-kn6xx"
Oct 30 21:49:55.315320 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:55.315301    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/87c44883-54d0-4a05-b94b-d3c2835a03e1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kn6xx\" (UID: \"87c44883-54d0-4a05-b94b-d3c2835a03e1\") " pod="openshift-insights/insights-runtime-extractor-kn6xx"
Oct 30 21:49:55.315499 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:55.315369    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/69e218ec-ab5e-4e09-8444-e53a710c53bf-console-oauth-config\") pod \"console-69874d4cdd-zhlp5\" (UID: \"69e218ec-ab5e-4e09-8444-e53a710c53bf\") " pod="openshift-console/console-69874d4cdd-zhlp5"
Oct 30 21:49:55.315499 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:55.315458    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdznq\" (UniqueName: \"kubernetes.io/projected/87c44883-54d0-4a05-b94b-d3c2835a03e1-kube-api-access-cdznq\") pod \"insights-runtime-extractor-kn6xx\" (UID: \"87c44883-54d0-4a05-b94b-d3c2835a03e1\") " pod="openshift-insights/insights-runtime-extractor-kn6xx"
Oct 30 21:49:55.315559 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:55.315528    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/87c44883-54d0-4a05-b94b-d3c2835a03e1-data-volume\") pod \"insights-runtime-extractor-kn6xx\" (UID: \"87c44883-54d0-4a05-b94b-d3c2835a03e1\") " pod="openshift-insights/insights-runtime-extractor-kn6xx"
Oct 30 21:49:55.315559 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:55.315543    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/69e218ec-ab5e-4e09-8444-e53a710c53bf-tmp\") pod \"console-69874d4cdd-zhlp5\" (UID: \"69e218ec-ab5e-4e09-8444-e53a710c53bf\") " pod="openshift-console/console-69874d4cdd-zhlp5"
Oct 30 21:49:55.315628 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:55.315586    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66wt8\" (UniqueName: \"kubernetes.io/projected/516573f9-4ada-4868-b885-f61b9640e6bf-kube-api-access-66wt8\") pod \"ingress-canary-hr227\" (UID: \"516573f9-4ada-4868-b885-f61b9640e6bf\") " pod="openshift-ingress-canary/ingress-canary-hr227"
Oct 30 21:49:55.315662 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:55.315623    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/69e218ec-ab5e-4e09-8444-e53a710c53bf-service-ca\") pod \"console-69874d4cdd-zhlp5\" (UID: \"69e218ec-ab5e-4e09-8444-e53a710c53bf\") " pod="openshift-console/console-69874d4cdd-zhlp5"
Oct 30 21:49:55.315711 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:55.315678    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/516573f9-4ada-4868-b885-f61b9640e6bf-cert\") pod \"ingress-canary-hr227\" (UID: \"516573f9-4ada-4868-b885-f61b9640e6bf\") " pod="openshift-ingress-canary/ingress-canary-hr227"
Oct 30 21:49:55.315763 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:55.315729    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/49ff8cb6-54de-4e8c-9a35-634f9a1ef29b-metrics-tls\") pod \"dns-default-fdmdz\" (UID: \"49ff8cb6-54de-4e8c-9a35-634f9a1ef29b\") " pod="openshift-dns/dns-default-fdmdz"
Oct 30 21:49:55.315798 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:55.315769    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/69e218ec-ab5e-4e09-8444-e53a710c53bf-oauth-serving-cert\") pod \"console-69874d4cdd-zhlp5\" (UID: \"69e218ec-ab5e-4e09-8444-e53a710c53bf\") " pod="openshift-console/console-69874d4cdd-zhlp5"
Oct 30 21:49:55.319715 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:55.319696    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/516573f9-4ada-4868-b885-f61b9640e6bf-cert\") pod \"ingress-canary-hr227\" (UID: \"516573f9-4ada-4868-b885-f61b9640e6bf\") " pod="openshift-ingress-canary/ingress-canary-hr227"
Oct 30 21:49:55.334142 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:55.334117    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Oct 30 21:49:55.334142 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:55.334118    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Oct 30 21:49:55.334142 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:55.334138    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 21:49:55.334440 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:55.334424    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 21:49:55.334524 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:55.334451    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 21:49:55.334524 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:55.334460    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 21:49:55.334524 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:55.334476    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Oct 30 21:49:55.334524 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:55.334485    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Oct 30 21:49:55.334670 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:55.334598    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 21:49:55.334848 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:55.334837    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Oct 30 21:49:55.337600 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:55.337585    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-8lg5r\""
Oct 30 21:49:55.337858 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:55.337842    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 21:49:55.337914 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:55.337889    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Oct 30 21:49:55.337914 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:55.337894    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Oct 30 21:49:55.337914 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:55.337905    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-bbh9g\""
Oct 30 21:49:55.338037 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:55.337949    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-52nbt\""
Oct 30 21:49:55.338037 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:55.337991    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 21:49:55.338037 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:55.338035    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-jlg2v\""
Oct 30 21:49:55.374970 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:55.374945    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66wt8\" (UniqueName: \"kubernetes.io/projected/516573f9-4ada-4868-b885-f61b9640e6bf-kube-api-access-66wt8\") pod \"ingress-canary-hr227\" (UID: \"516573f9-4ada-4868-b885-f61b9640e6bf\") " pod="openshift-ingress-canary/ingress-canary-hr227"
Oct 30 21:49:55.416582 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:55.416558    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/87c44883-54d0-4a05-b94b-d3c2835a03e1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kn6xx\" (UID: \"87c44883-54d0-4a05-b94b-d3c2835a03e1\") " pod="openshift-insights/insights-runtime-extractor-kn6xx"
Oct 30 21:49:55.416682 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:55.416585    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/49ff8cb6-54de-4e8c-9a35-634f9a1ef29b-config-volume\") pod \"dns-default-fdmdz\" (UID: \"49ff8cb6-54de-4e8c-9a35-634f9a1ef29b\") " pod="openshift-dns/dns-default-fdmdz"
Oct 30 21:49:55.416682 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:55.416608    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/69e218ec-ab5e-4e09-8444-e53a710c53bf-console-serving-cert\") pod \"console-69874d4cdd-zhlp5\" (UID: \"69e218ec-ab5e-4e09-8444-e53a710c53bf\") " pod="openshift-console/console-69874d4cdd-zhlp5"
Oct 30 21:49:55.416682 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:55.416628    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49ff8cb6-54de-4e8c-9a35-634f9a1ef29b-tmp-dir\") pod \"dns-default-fdmdz\" (UID: \"49ff8cb6-54de-4e8c-9a35-634f9a1ef29b\") " pod="openshift-dns/dns-default-fdmdz"
Oct 30 21:49:55.416682 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:55.416643    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gv6wf\" (UniqueName: \"kubernetes.io/projected/69e218ec-ab5e-4e09-8444-e53a710c53bf-kube-api-access-gv6wf\") pod \"console-69874d4cdd-zhlp5\" (UID: \"69e218ec-ab5e-4e09-8444-e53a710c53bf\") " pod="openshift-console/console-69874d4cdd-zhlp5"
Oct 30 21:49:55.416682 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:55.416663    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7qxb\" (UniqueName: \"kubernetes.io/projected/49ff8cb6-54de-4e8c-9a35-634f9a1ef29b-kube-api-access-z7qxb\") pod \"dns-default-fdmdz\" (UID: \"49ff8cb6-54de-4e8c-9a35-634f9a1ef29b\") " pod="openshift-dns/dns-default-fdmdz"
Oct 30 21:49:55.416911 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:55.416689    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/69e218ec-ab5e-4e09-8444-e53a710c53bf-console-config\") pod \"console-69874d4cdd-zhlp5\" (UID: \"69e218ec-ab5e-4e09-8444-e53a710c53bf\") " pod="openshift-console/console-69874d4cdd-zhlp5"
Oct 30 21:49:55.416911 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:55.416741    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/87c44883-54d0-4a05-b94b-d3c2835a03e1-crio-socket\") pod \"insights-runtime-extractor-kn6xx\" (UID: \"87c44883-54d0-4a05-b94b-d3c2835a03e1\") " pod="openshift-insights/insights-runtime-extractor-kn6xx"
Oct 30 21:49:55.417045 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:55.417025    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49ff8cb6-54de-4e8c-9a35-634f9a1ef29b-tmp-dir\") pod \"dns-default-fdmdz\" (UID: \"49ff8cb6-54de-4e8c-9a35-634f9a1ef29b\") " pod="openshift-dns/dns-default-fdmdz"
Oct 30 21:49:55.417115 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:55.417026    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/87c44883-54d0-4a05-b94b-d3c2835a03e1-crio-socket\") pod \"insights-runtime-extractor-kn6xx\" (UID: \"87c44883-54d0-4a05-b94b-d3c2835a03e1\") " pod="openshift-insights/insights-runtime-extractor-kn6xx"
Oct 30 21:49:55.417115 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:55.417095    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/87c44883-54d0-4a05-b94b-d3c2835a03e1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kn6xx\" (UID: \"87c44883-54d0-4a05-b94b-d3c2835a03e1\") " pod="openshift-insights/insights-runtime-extractor-kn6xx"
Oct 30 21:49:55.417207 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:55.417148    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/69e218ec-ab5e-4e09-8444-e53a710c53bf-console-oauth-config\") pod \"console-69874d4cdd-zhlp5\" (UID: \"69e218ec-ab5e-4e09-8444-e53a710c53bf\") " pod="openshift-console/console-69874d4cdd-zhlp5"
Oct 30 21:49:55.417207 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:55.417181    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdznq\" (UniqueName: \"kubernetes.io/projected/87c44883-54d0-4a05-b94b-d3c2835a03e1-kube-api-access-cdznq\") pod \"insights-runtime-extractor-kn6xx\" (UID: \"87c44883-54d0-4a05-b94b-d3c2835a03e1\") " pod="openshift-insights/insights-runtime-extractor-kn6xx"
Oct 30 21:49:55.417309 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:55.417216    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/87c44883-54d0-4a05-b94b-d3c2835a03e1-data-volume\") pod \"insights-runtime-extractor-kn6xx\" (UID: \"87c44883-54d0-4a05-b94b-d3c2835a03e1\") " pod="openshift-insights/insights-runtime-extractor-kn6xx"
Oct 30 21:49:55.417309 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:55.417239    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/69e218ec-ab5e-4e09-8444-e53a710c53bf-tmp\") pod \"console-69874d4cdd-zhlp5\" (UID: \"69e218ec-ab5e-4e09-8444-e53a710c53bf\") " pod="openshift-console/console-69874d4cdd-zhlp5"
Oct 30 21:49:55.417309 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:55.417251    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/49ff8cb6-54de-4e8c-9a35-634f9a1ef29b-config-volume\") pod \"dns-default-fdmdz\" (UID: \"49ff8cb6-54de-4e8c-9a35-634f9a1ef29b\") " pod="openshift-dns/dns-default-fdmdz"
Oct 30 21:49:55.417309 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:55.417282    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/69e218ec-ab5e-4e09-8444-e53a710c53bf-service-ca\") pod \"console-69874d4cdd-zhlp5\" (UID: \"69e218ec-ab5e-4e09-8444-e53a710c53bf\") " pod="openshift-console/console-69874d4cdd-zhlp5"
Oct 30 21:49:55.417514 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:55.417326    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/49ff8cb6-54de-4e8c-9a35-634f9a1ef29b-metrics-tls\") pod \"dns-default-fdmdz\" (UID: \"49ff8cb6-54de-4e8c-9a35-634f9a1ef29b\") " pod="openshift-dns/dns-default-fdmdz"
Oct 30 21:49:55.417514 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:55.417352    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/69e218ec-ab5e-4e09-8444-e53a710c53bf-oauth-serving-cert\") pod \"console-69874d4cdd-zhlp5\" (UID: \"69e218ec-ab5e-4e09-8444-e53a710c53bf\") " pod="openshift-console/console-69874d4cdd-zhlp5"
Oct 30 21:49:55.417514 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:55.417493    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/69e218ec-ab5e-4e09-8444-e53a710c53bf-console-config\") pod \"console-69874d4cdd-zhlp5\" (UID: \"69e218ec-ab5e-4e09-8444-e53a710c53bf\") " pod="openshift-console/console-69874d4cdd-zhlp5"
Oct 30 21:49:55.417669 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:55.417635    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/87c44883-54d0-4a05-b94b-d3c2835a03e1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kn6xx\" (UID: \"87c44883-54d0-4a05-b94b-d3c2835a03e1\") " pod="openshift-insights/insights-runtime-extractor-kn6xx"
Oct 30 21:49:55.417761 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:55.417740    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/69e218ec-ab5e-4e09-8444-e53a710c53bf-tmp\") pod \"console-69874d4cdd-zhlp5\" (UID: \"69e218ec-ab5e-4e09-8444-e53a710c53bf\") " pod="openshift-console/console-69874d4cdd-zhlp5"
Oct 30 21:49:55.417761 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:55.417741    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/87c44883-54d0-4a05-b94b-d3c2835a03e1-data-volume\") pod \"insights-runtime-extractor-kn6xx\" (UID: \"87c44883-54d0-4a05-b94b-d3c2835a03e1\") " pod="openshift-insights/insights-runtime-extractor-kn6xx"
Oct 30 21:49:55.417971 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:55.417954    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/69e218ec-ab5e-4e09-8444-e53a710c53bf-oauth-serving-cert\") pod \"console-69874d4cdd-zhlp5\" (UID: \"69e218ec-ab5e-4e09-8444-e53a710c53bf\") " pod="openshift-console/console-69874d4cdd-zhlp5"
Oct 30 21:49:55.418246 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:55.418223    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/69e218ec-ab5e-4e09-8444-e53a710c53bf-service-ca\") pod \"console-69874d4cdd-zhlp5\" (UID: \"69e218ec-ab5e-4e09-8444-e53a710c53bf\") " pod="openshift-console/console-69874d4cdd-zhlp5"
Oct 30 21:49:55.419234 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:55.419216    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/69e218ec-ab5e-4e09-8444-e53a710c53bf-console-serving-cert\") pod \"console-69874d4cdd-zhlp5\" (UID: \"69e218ec-ab5e-4e09-8444-e53a710c53bf\") " pod="openshift-console/console-69874d4cdd-zhlp5"
Oct 30 21:49:55.419316 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:55.419256    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/87c44883-54d0-4a05-b94b-d3c2835a03e1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kn6xx\" (UID: \"87c44883-54d0-4a05-b94b-d3c2835a03e1\") " pod="openshift-insights/insights-runtime-extractor-kn6xx"
Oct 30 21:49:55.419642 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:55.419627    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/69e218ec-ab5e-4e09-8444-e53a710c53bf-console-oauth-config\") pod \"console-69874d4cdd-zhlp5\" (UID: \"69e218ec-ab5e-4e09-8444-e53a710c53bf\") " pod="openshift-console/console-69874d4cdd-zhlp5"
Oct 30 21:49:55.419890 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:55.419875    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/49ff8cb6-54de-4e8c-9a35-634f9a1ef29b-metrics-tls\") pod \"dns-default-fdmdz\" (UID: \"49ff8cb6-54de-4e8c-9a35-634f9a1ef29b\") " pod="openshift-dns/dns-default-fdmdz"
Oct 30 21:49:55.476333 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:55.476275    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdznq\" (UniqueName: \"kubernetes.io/projected/87c44883-54d0-4a05-b94b-d3c2835a03e1-kube-api-access-cdznq\") pod \"insights-runtime-extractor-kn6xx\" (UID: \"87c44883-54d0-4a05-b94b-d3c2835a03e1\") " pod="openshift-insights/insights-runtime-extractor-kn6xx"
Oct 30 21:49:55.476443 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:55.476359    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gv6wf\" (UniqueName: \"kubernetes.io/projected/69e218ec-ab5e-4e09-8444-e53a710c53bf-kube-api-access-gv6wf\") pod \"console-69874d4cdd-zhlp5\" (UID: \"69e218ec-ab5e-4e09-8444-e53a710c53bf\") " pod="openshift-console/console-69874d4cdd-zhlp5"
Oct 30 21:49:55.476491 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:55.476454    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7qxb\" (UniqueName: \"kubernetes.io/projected/49ff8cb6-54de-4e8c-9a35-634f9a1ef29b-kube-api-access-z7qxb\") pod \"dns-default-fdmdz\" (UID: \"49ff8cb6-54de-4e8c-9a35-634f9a1ef29b\") " pod="openshift-dns/dns-default-fdmdz"
Oct 30 21:49:55.480469 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:55.480445    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xbh9m"
Oct 30 21:49:55.492890 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:55.492874    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hr227"
Oct 30 21:49:55.530727 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:55.530699    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 21:49:55.530879 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:55.530748    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9bjlv\""
Oct 30 21:49:55.542002 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:55.541979    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-kn6xx"
Oct 30 21:49:55.546619 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:55.546597    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-fdmdz"
Oct 30 21:49:55.552254 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:55.550771    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69874d4cdd-zhlp5"
Oct 30 21:49:55.865710 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:55.864331    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hr227"]
Oct 30 21:49:55.914870 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:55.914836    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-kn6xx"]
Oct 30 21:49:55.918308 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:55.918283    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87c44883_54d0_4a05_b94b_d3c2835a03e1.slice/crio-b8fd19e59bf3f63ce8b861120a1c5a46871e91627d90f3725696a99fc1e6f4f5 WatchSource:0}: Error finding container b8fd19e59bf3f63ce8b861120a1c5a46871e91627d90f3725696a99fc1e6f4f5: Status 404 returned error can't find the container with id b8fd19e59bf3f63ce8b861120a1c5a46871e91627d90f3725696a99fc1e6f4f5
Oct 30 21:49:55.963068 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:55.963044    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-69874d4cdd-zhlp5"]
Oct 30 21:49:55.965483 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:55.965459    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-fdmdz"]
Oct 30 21:49:55.966999 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:55.966977    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69e218ec_ab5e_4e09_8444_e53a710c53bf.slice/crio-ca41b3353f4cff5c25e5091dd9731c19f4a1577f6aceaf86e9b91db22ba7f5e1 WatchSource:0}: Error finding container ca41b3353f4cff5c25e5091dd9731c19f4a1577f6aceaf86e9b91db22ba7f5e1: Status 404 returned error can't find the container with id ca41b3353f4cff5c25e5091dd9731c19f4a1577f6aceaf86e9b91db22ba7f5e1
Oct 30 21:49:55.968434 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:55.968391    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49ff8cb6_54de_4e8c_9a35_634f9a1ef29b.slice/crio-929ab3025f8a48c7f0741da506a9447ead41f7e897a54256d70954ade722c613 WatchSource:0}: Error finding container 929ab3025f8a48c7f0741da506a9447ead41f7e897a54256d70954ade722c613: Status 404 returned error can't find the container with id 929ab3025f8a48c7f0741da506a9447ead41f7e897a54256d70954ade722c613
Oct 30 21:49:56.480371 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:56.480337    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cck7d"
Oct 30 21:49:56.481066 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:56.480337    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w9tx8"
Oct 30 21:49:56.530534 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:56.530503    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 21:49:56.530638 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:56.530576    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 21:49:56.530705 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:56.530665    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 21:49:56.530705 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:56.530671    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-9mgc8\""
Oct 30 21:49:56.530805 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:56.530765    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-jbflf\""
Oct 30 21:49:56.708795 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:56.708764    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hr227" event={"ID":"516573f9-4ada-4868-b885-f61b9640e6bf","Type":"ContainerStarted","Data":"da9e0ba8a0e86180f7612fe2a827ab57b9c0afcdf1a087ac9140a2b5de077636"}
Oct 30 21:49:56.710267 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:56.710239    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kn6xx" event={"ID":"87c44883-54d0-4a05-b94b-d3c2835a03e1","Type":"ContainerStarted","Data":"c32b6741c6eb32619e8cd0b10af8e50caca1572926caf35bcd3dd7b8685ecdc8"}
Oct 30 21:49:56.710372 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:56.710297    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kn6xx" event={"ID":"87c44883-54d0-4a05-b94b-d3c2835a03e1","Type":"ContainerStarted","Data":"b8fd19e59bf3f63ce8b861120a1c5a46871e91627d90f3725696a99fc1e6f4f5"}
Oct 30 21:49:56.711428 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:56.711386    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69874d4cdd-zhlp5" event={"ID":"69e218ec-ab5e-4e09-8444-e53a710c53bf","Type":"ContainerStarted","Data":"ca41b3353f4cff5c25e5091dd9731c19f4a1577f6aceaf86e9b91db22ba7f5e1"}
Oct 30 21:49:56.712429 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:56.712389    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fdmdz" event={"ID":"49ff8cb6-54de-4e8c-9a35-634f9a1ef29b","Type":"ContainerStarted","Data":"929ab3025f8a48c7f0741da506a9447ead41f7e897a54256d70954ade722c613"}
Oct 30 21:49:57.723172 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:57.723098    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kn6xx" event={"ID":"87c44883-54d0-4a05-b94b-d3c2835a03e1","Type":"ContainerStarted","Data":"7caf62e46c0a56f106127c5fbb414152ed129881787850a5d54595f0d293b5b9"}
Oct 30 21:49:58.458609 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:58.458576    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-vcj4q"]
Oct 30 21:49:58.461114 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:58.461090    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-vcj4q"
Oct 30 21:49:58.539163 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:58.539131    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/67b50e2a-4d01-4d5f-a5d1-7e80ee62642a-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-vcj4q\" (UID: \"67b50e2a-4d01-4d5f-a5d1-7e80ee62642a\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-vcj4q"
Oct 30 21:49:58.539163 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:58.539167    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fn582\" (UniqueName: \"kubernetes.io/projected/67b50e2a-4d01-4d5f-a5d1-7e80ee62642a-kube-api-access-fn582\") pod \"prometheus-operator-9cd89b74c-vcj4q\" (UID: \"67b50e2a-4d01-4d5f-a5d1-7e80ee62642a\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-vcj4q"
Oct 30 21:49:58.539420 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:58.539335    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/67b50e2a-4d01-4d5f-a5d1-7e80ee62642a-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-vcj4q\" (UID: \"67b50e2a-4d01-4d5f-a5d1-7e80ee62642a\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-vcj4q"
Oct 30 21:49:58.539420 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:58.539372    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/67b50e2a-4d01-4d5f-a5d1-7e80ee62642a-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-vcj4q\" (UID: \"67b50e2a-4d01-4d5f-a5d1-7e80ee62642a\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-vcj4q"
Oct 30 21:49:58.560802 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:58.560768    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 21:49:58.560802 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:58.560769    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Oct 30 21:49:58.561253 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:58.561234    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 21:49:58.561371 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:58.561286    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Oct 30 21:49:58.561462 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:58.561420    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 30 21:49:58.608044 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:58.608007    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-nzxnt\""
Oct 30 21:49:58.609495 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:58.609469    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-vcj4q"]
Oct 30 21:49:58.641023 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:58.640380    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/67b50e2a-4d01-4d5f-a5d1-7e80ee62642a-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-vcj4q\" (UID: \"67b50e2a-4d01-4d5f-a5d1-7e80ee62642a\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-vcj4q"
Oct 30 21:49:58.641023 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:58.640457    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fn582\" (UniqueName: \"kubernetes.io/projected/67b50e2a-4d01-4d5f-a5d1-7e80ee62642a-kube-api-access-fn582\") pod \"prometheus-operator-9cd89b74c-vcj4q\" (UID: \"67b50e2a-4d01-4d5f-a5d1-7e80ee62642a\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-vcj4q"
Oct 30 21:49:58.641023 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:58.640551    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/67b50e2a-4d01-4d5f-a5d1-7e80ee62642a-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-vcj4q\" (UID: \"67b50e2a-4d01-4d5f-a5d1-7e80ee62642a\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-vcj4q"
Oct 30 21:49:58.641023 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:58.640576    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/67b50e2a-4d01-4d5f-a5d1-7e80ee62642a-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-vcj4q\" (UID: \"67b50e2a-4d01-4d5f-a5d1-7e80ee62642a\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-vcj4q"
Oct 30 21:49:58.641293 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:58.641267    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/67b50e2a-4d01-4d5f-a5d1-7e80ee62642a-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-vcj4q\" (UID: \"67b50e2a-4d01-4d5f-a5d1-7e80ee62642a\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-vcj4q"
Oct 30 21:49:58.645180 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:58.645153    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/67b50e2a-4d01-4d5f-a5d1-7e80ee62642a-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-vcj4q\" (UID: \"67b50e2a-4d01-4d5f-a5d1-7e80ee62642a\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-vcj4q"
Oct 30 21:49:58.645893 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:58.645869    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/67b50e2a-4d01-4d5f-a5d1-7e80ee62642a-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-vcj4q\" (UID: \"67b50e2a-4d01-4d5f-a5d1-7e80ee62642a\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-vcj4q"
Oct 30 21:49:58.701900 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:58.701874    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fn582\" (UniqueName: \"kubernetes.io/projected/67b50e2a-4d01-4d5f-a5d1-7e80ee62642a-kube-api-access-fn582\") pod \"prometheus-operator-9cd89b74c-vcj4q\" (UID: \"67b50e2a-4d01-4d5f-a5d1-7e80ee62642a\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-vcj4q"
Oct 30 21:49:58.727324 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:58.727293    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fdmdz" event={"ID":"49ff8cb6-54de-4e8c-9a35-634f9a1ef29b","Type":"ContainerStarted","Data":"ea3042a43a496ea4e5281f4a33003259ccf0084a30ad8b619769b6a841cd0297"}
Oct 30 21:49:58.775219 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:58.775172    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-vcj4q"
Oct 30 21:49:59.176062 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:59.176025    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-vcj4q"]
Oct 30 21:49:59.197694 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:49:59.197047    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67b50e2a_4d01_4d5f_a5d1_7e80ee62642a.slice/crio-4768f1ea299aaff0c7dc0db8d94ec8e94771f4682b4b37ea448a32a4eaa05704 WatchSource:0}: Error finding container 4768f1ea299aaff0c7dc0db8d94ec8e94771f4682b4b37ea448a32a4eaa05704: Status 404 returned error can't find the container with id 4768f1ea299aaff0c7dc0db8d94ec8e94771f4682b4b37ea448a32a4eaa05704
Oct 30 21:49:59.731540 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:59.731506    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-vcj4q" event={"ID":"67b50e2a-4d01-4d5f-a5d1-7e80ee62642a","Type":"ContainerStarted","Data":"4768f1ea299aaff0c7dc0db8d94ec8e94771f4682b4b37ea448a32a4eaa05704"}
Oct 30 21:49:59.733833 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:59.733803    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fdmdz" event={"ID":"49ff8cb6-54de-4e8c-9a35-634f9a1ef29b","Type":"ContainerStarted","Data":"82936a1cb72fed53755cf70a7e0b77c7b9db2b0dad65a32caa315ab297c2135b"}
Oct 30 21:49:59.733972 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:59.733954    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-fdmdz"
Oct 30 21:49:59.850499 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:49:59.850431    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-fdmdz" podStartSLOduration=2.204438369 podStartE2EDuration="4.850390155s" totalImagesPullingTime="2.645951786s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:49:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:59.849366512 +0000 UTC m=+50.971991251" watchObservedRunningTime="2026-10-30 21:49:59.850390155 +0000 UTC m=+50.973014884"
Oct 30 21:50:00.739658 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:00.739614    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hr227" event={"ID":"516573f9-4ada-4868-b885-f61b9640e6bf","Type":"ContainerStarted","Data":"8e22635c0f4cbe571c843b683eae774d5660ca7e26c654f4311c35be595034c5"}
Oct 30 21:50:00.742104 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:00.742073    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kn6xx" event={"ID":"87c44883-54d0-4a05-b94b-d3c2835a03e1","Type":"ContainerStarted","Data":"9b9fb508be8d6542ca8bae7e1711a822c9752465c69d5ad50ccebe047861a37b"}
Oct 30 21:50:00.743772 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:00.743720    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69874d4cdd-zhlp5" event={"ID":"69e218ec-ab5e-4e09-8444-e53a710c53bf","Type":"ContainerStarted","Data":"510dece2cd11b7771e421f49c46269e9e21714206ddf570252332975593296c6"}
Oct 30 21:50:00.850795 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:00.850735    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-hr227" podStartSLOduration=1.990010294 podStartE2EDuration="5.850716715s" totalImagesPullingTime="3.860706421s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:49:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:50:00.850531097 +0000 UTC m=+51.973155827" watchObservedRunningTime="2026-10-30 21:50:00.850716715 +0000 UTC m=+51.973341445"
Oct 30 21:50:00.965981 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:00.965931    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-69874d4cdd-zhlp5" podStartSLOduration=1.9652147420000001 podStartE2EDuration="5.965914765s" totalImagesPullingTime="4.000700023s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:49:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:50:00.963384062 +0000 UTC m=+52.086008814" watchObservedRunningTime="2026-10-30 21:50:00.965914765 +0000 UTC m=+52.088539547"
Oct 30 21:50:01.077702 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:01.077659    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-kn6xx" podStartSLOduration=2.3038186019999998 podStartE2EDuration="6.077644281s" totalImagesPullingTime="3.773825679s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:49:55 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:50:01.077222687 +0000 UTC m=+52.199847415" watchObservedRunningTime="2026-10-30 21:50:01.077644281 +0000 UTC m=+52.200269010"
Oct 30 21:50:01.748626 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:01.748591    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-vcj4q" event={"ID":"67b50e2a-4d01-4d5f-a5d1-7e80ee62642a","Type":"ContainerStarted","Data":"22265062683a6d53f2c0f502c23ea3d025443082c05b9a3acc8d68dbd700772d"}
Oct 30 21:50:01.748988 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:01.748640    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-vcj4q" event={"ID":"67b50e2a-4d01-4d5f-a5d1-7e80ee62642a","Type":"ContainerStarted","Data":"6df5ccc1be9d17b71a01679da8abacf177500b1f210b32b2040dd80a9d162012"}
Oct 30 21:50:01.861427 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:01.861314    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-vcj4q" podStartSLOduration=2.087919409 podStartE2EDuration="3.861295529s" totalImagesPullingTime="1.77337612s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:49:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:50:01.860866018 +0000 UTC m=+52.983490747" watchObservedRunningTime="2026-10-30 21:50:01.861295529 +0000 UTC m=+52.983920253"
Oct 30 21:50:05.145460 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:05.145429    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-pvl2l"]
Oct 30 21:50:05.147891 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:05.147867    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pvl2l"
Oct 30 21:50:05.190310 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:05.190283    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/dec1cd54-fec6-4c12-9d37-a1b0b1a7ffc0-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-pvl2l\" (UID: \"dec1cd54-fec6-4c12-9d37-a1b0b1a7ffc0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pvl2l"
Oct 30 21:50:05.190427 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:05.190329    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/dec1cd54-fec6-4c12-9d37-a1b0b1a7ffc0-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-pvl2l\" (UID: \"dec1cd54-fec6-4c12-9d37-a1b0b1a7ffc0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pvl2l"
Oct 30 21:50:05.190472 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:05.190445    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dec1cd54-fec6-4c12-9d37-a1b0b1a7ffc0-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-pvl2l\" (UID: \"dec1cd54-fec6-4c12-9d37-a1b0b1a7ffc0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pvl2l"
Oct 30 21:50:05.190506 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:05.190494    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/dec1cd54-fec6-4c12-9d37-a1b0b1a7ffc0-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-pvl2l\" (UID: \"dec1cd54-fec6-4c12-9d37-a1b0b1a7ffc0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pvl2l"
Oct 30 21:50:05.190538 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:05.190514    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dec1cd54-fec6-4c12-9d37-a1b0b1a7ffc0-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-pvl2l\" (UID: \"dec1cd54-fec6-4c12-9d37-a1b0b1a7ffc0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pvl2l"
Oct 30 21:50:05.190572 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:05.190551    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvbdq\" (UniqueName: \"kubernetes.io/projected/dec1cd54-fec6-4c12-9d37-a1b0b1a7ffc0-kube-api-access-wvbdq\") pod \"kube-state-metrics-5dc8d7d89b-pvl2l\" (UID: \"dec1cd54-fec6-4c12-9d37-a1b0b1a7ffc0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pvl2l"
Oct 30 21:50:05.192487 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:05.192470    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-7j5j6"]
Oct 30 21:50:05.194842 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:05.194829    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7j5j6"
Oct 30 21:50:05.241807 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:05.241778    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Oct 30 21:50:05.244184 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:05.244161    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-4qz2z\""
Oct 30 21:50:05.244319 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:05.244301    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Oct 30 21:50:05.244380 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:05.244301    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Oct 30 21:50:05.245421 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:05.245391    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-pvl2l"]
Oct 30 21:50:05.288869 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:05.288836    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 30 21:50:05.291113 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:05.291086    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f938a93b-29dd-4e7d-b8e9-444df68acf03-sys\") pod \"node-exporter-7j5j6\" (UID: \"f938a93b-29dd-4e7d-b8e9-444df68acf03\") " pod="openshift-monitoring/node-exporter-7j5j6"
Oct 30 21:50:05.291211 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:05.291120    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/dec1cd54-fec6-4c12-9d37-a1b0b1a7ffc0-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-pvl2l\" (UID: \"dec1cd54-fec6-4c12-9d37-a1b0b1a7ffc0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pvl2l"
Oct 30 21:50:05.291211 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:05.291148    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f938a93b-29dd-4e7d-b8e9-444df68acf03-root\") pod \"node-exporter-7j5j6\" (UID: \"f938a93b-29dd-4e7d-b8e9-444df68acf03\") " pod="openshift-monitoring/node-exporter-7j5j6"
Oct 30 21:50:05.291331 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:05.291233    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/dec1cd54-fec6-4c12-9d37-a1b0b1a7ffc0-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-pvl2l\" (UID: \"dec1cd54-fec6-4c12-9d37-a1b0b1a7ffc0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pvl2l"
Oct 30 21:50:05.291331 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:05.291264    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dec1cd54-fec6-4c12-9d37-a1b0b1a7ffc0-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-pvl2l\" (UID: \"dec1cd54-fec6-4c12-9d37-a1b0b1a7ffc0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pvl2l"
Oct 30 21:50:05.291331 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:05.291287    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f938a93b-29dd-4e7d-b8e9-444df68acf03-node-exporter-accelerators-collector-config\") pod \"node-exporter-7j5j6\" (UID: \"f938a93b-29dd-4e7d-b8e9-444df68acf03\") " pod="openshift-monitoring/node-exporter-7j5j6"
Oct 30 21:50:05.291331 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:05.291308    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f938a93b-29dd-4e7d-b8e9-444df68acf03-node-exporter-tls\") pod \"node-exporter-7j5j6\" (UID: \"f938a93b-29dd-4e7d-b8e9-444df68acf03\") " pod="openshift-monitoring/node-exporter-7j5j6"
Oct 30 21:50:05.291331 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:05.291327    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f938a93b-29dd-4e7d-b8e9-444df68acf03-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7j5j6\" (UID: \"f938a93b-29dd-4e7d-b8e9-444df68acf03\") " pod="openshift-monitoring/node-exporter-7j5j6"
Oct 30 21:50:05.291615 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:05.291345    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/dec1cd54-fec6-4c12-9d37-a1b0b1a7ffc0-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-pvl2l\" (UID: \"dec1cd54-fec6-4c12-9d37-a1b0b1a7ffc0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pvl2l"
Oct 30 21:50:05.291615 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:05.291412    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f938a93b-29dd-4e7d-b8e9-444df68acf03-node-exporter-wtmp\") pod \"node-exporter-7j5j6\" (UID: \"f938a93b-29dd-4e7d-b8e9-444df68acf03\") " pod="openshift-monitoring/node-exporter-7j5j6"
Oct 30 21:50:05.291615 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:05.291577    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f938a93b-29dd-4e7d-b8e9-444df68acf03-node-exporter-textfile\") pod \"node-exporter-7j5j6\" (UID: \"f938a93b-29dd-4e7d-b8e9-444df68acf03\") " pod="openshift-monitoring/node-exporter-7j5j6"
Oct 30 21:50:05.291757 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:05.291615    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvbdq\" (UniqueName: \"kubernetes.io/projected/dec1cd54-fec6-4c12-9d37-a1b0b1a7ffc0-kube-api-access-wvbdq\") pod \"kube-state-metrics-5dc8d7d89b-pvl2l\" (UID: \"dec1cd54-fec6-4c12-9d37-a1b0b1a7ffc0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pvl2l"
Oct 30 21:50:05.291757 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:05.291653    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jt5t\" (UniqueName: \"kubernetes.io/projected/f938a93b-29dd-4e7d-b8e9-444df68acf03-kube-api-access-6jt5t\") pod \"node-exporter-7j5j6\" (UID: \"f938a93b-29dd-4e7d-b8e9-444df68acf03\") " pod="openshift-monitoring/node-exporter-7j5j6"
Oct 30 21:50:05.291757 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:05.291718    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dec1cd54-fec6-4c12-9d37-a1b0b1a7ffc0-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-pvl2l\" (UID: \"dec1cd54-fec6-4c12-9d37-a1b0b1a7ffc0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pvl2l"
Oct 30 21:50:05.291757 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:05.291742    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/dec1cd54-fec6-4c12-9d37-a1b0b1a7ffc0-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-pvl2l\" (UID: \"dec1cd54-fec6-4c12-9d37-a1b0b1a7ffc0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pvl2l"
Oct 30 21:50:05.291972 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:05.291756    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f938a93b-29dd-4e7d-b8e9-444df68acf03-metrics-client-ca\") pod \"node-exporter-7j5j6\" (UID: \"f938a93b-29dd-4e7d-b8e9-444df68acf03\") " pod="openshift-monitoring/node-exporter-7j5j6"
Oct 30 21:50:05.292026 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:05.292008    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/dec1cd54-fec6-4c12-9d37-a1b0b1a7ffc0-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-pvl2l\" (UID: \"dec1cd54-fec6-4c12-9d37-a1b0b1a7ffc0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pvl2l"
Oct 30 21:50:05.292101 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:05.292081    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dec1cd54-fec6-4c12-9d37-a1b0b1a7ffc0-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-pvl2l\" (UID: \"dec1cd54-fec6-4c12-9d37-a1b0b1a7ffc0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pvl2l"
Oct 30 21:50:05.293826 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:05.293806    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/dec1cd54-fec6-4c12-9d37-a1b0b1a7ffc0-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-pvl2l\" (UID: \"dec1cd54-fec6-4c12-9d37-a1b0b1a7ffc0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pvl2l"
Oct 30 21:50:05.293983 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:05.293969    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dec1cd54-fec6-4c12-9d37-a1b0b1a7ffc0-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-pvl2l\" (UID: \"dec1cd54-fec6-4c12-9d37-a1b0b1a7ffc0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pvl2l"
Oct 30 21:50:05.296569 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:05.296556    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 30 21:50:05.296838 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:05.296821    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-dhl5t\""
Oct 30 21:50:05.296920 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:05.296909    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 30 21:50:05.350165 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:05.350138    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvbdq\" (UniqueName: \"kubernetes.io/projected/dec1cd54-fec6-4c12-9d37-a1b0b1a7ffc0-kube-api-access-wvbdq\") pod \"kube-state-metrics-5dc8d7d89b-pvl2l\" (UID: \"dec1cd54-fec6-4c12-9d37-a1b0b1a7ffc0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pvl2l"
Oct 30 21:50:05.392780 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:05.392749    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f938a93b-29dd-4e7d-b8e9-444df68acf03-node-exporter-accelerators-collector-config\") pod \"node-exporter-7j5j6\" (UID: \"f938a93b-29dd-4e7d-b8e9-444df68acf03\") " pod="openshift-monitoring/node-exporter-7j5j6"
Oct 30 21:50:05.392885 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:05.392789    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f938a93b-29dd-4e7d-b8e9-444df68acf03-node-exporter-tls\") pod \"node-exporter-7j5j6\" (UID: \"f938a93b-29dd-4e7d-b8e9-444df68acf03\") " pod="openshift-monitoring/node-exporter-7j5j6"
Oct 30 21:50:05.392885 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:05.392818    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f938a93b-29dd-4e7d-b8e9-444df68acf03-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7j5j6\" (UID: \"f938a93b-29dd-4e7d-b8e9-444df68acf03\") " pod="openshift-monitoring/node-exporter-7j5j6"
Oct 30 21:50:05.392885 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:05.392848    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f938a93b-29dd-4e7d-b8e9-444df68acf03-node-exporter-wtmp\") pod \"node-exporter-7j5j6\" (UID: \"f938a93b-29dd-4e7d-b8e9-444df68acf03\") " pod="openshift-monitoring/node-exporter-7j5j6"
Oct 30 21:50:05.393048 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:05.392899    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f938a93b-29dd-4e7d-b8e9-444df68acf03-node-exporter-textfile\") pod \"node-exporter-7j5j6\" (UID: \"f938a93b-29dd-4e7d-b8e9-444df68acf03\") " pod="openshift-monitoring/node-exporter-7j5j6"
Oct 30 21:50:05.393048 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:05.392938    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jt5t\" (UniqueName: \"kubernetes.io/projected/f938a93b-29dd-4e7d-b8e9-444df68acf03-kube-api-access-6jt5t\") pod \"node-exporter-7j5j6\" (UID: \"f938a93b-29dd-4e7d-b8e9-444df68acf03\") " pod="openshift-monitoring/node-exporter-7j5j6"
Oct 30 21:50:05.393048 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:05.392975    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f938a93b-29dd-4e7d-b8e9-444df68acf03-metrics-client-ca\") pod \"node-exporter-7j5j6\" (UID: \"f938a93b-29dd-4e7d-b8e9-444df68acf03\") " pod="openshift-monitoring/node-exporter-7j5j6"
Oct 30 21:50:05.393048 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:05.393001    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f938a93b-29dd-4e7d-b8e9-444df68acf03-sys\") pod \"node-exporter-7j5j6\" (UID: \"f938a93b-29dd-4e7d-b8e9-444df68acf03\") " pod="openshift-monitoring/node-exporter-7j5j6"
Oct 30 21:50:05.393048 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:05.393043    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f938a93b-29dd-4e7d-b8e9-444df68acf03-root\") pod \"node-exporter-7j5j6\" (UID: \"f938a93b-29dd-4e7d-b8e9-444df68acf03\") " pod="openshift-monitoring/node-exporter-7j5j6"
Oct 30 21:50:05.393268 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:05.393046    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f938a93b-29dd-4e7d-b8e9-444df68acf03-node-exporter-wtmp\") pod \"node-exporter-7j5j6\" (UID: \"f938a93b-29dd-4e7d-b8e9-444df68acf03\") " pod="openshift-monitoring/node-exporter-7j5j6"
Oct 30 21:50:05.393268 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:05.393109    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f938a93b-29dd-4e7d-b8e9-444df68acf03-root\") pod \"node-exporter-7j5j6\" (UID: \"f938a93b-29dd-4e7d-b8e9-444df68acf03\") " pod="openshift-monitoring/node-exporter-7j5j6"
Oct 30 21:50:05.393268 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:05.393192    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f938a93b-29dd-4e7d-b8e9-444df68acf03-sys\") pod \"node-exporter-7j5j6\" (UID: \"f938a93b-29dd-4e7d-b8e9-444df68acf03\") " pod="openshift-monitoring/node-exporter-7j5j6"
Oct 30 21:50:05.393268 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:05.393252    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f938a93b-29dd-4e7d-b8e9-444df68acf03-node-exporter-textfile\") pod \"node-exporter-7j5j6\" (UID: \"f938a93b-29dd-4e7d-b8e9-444df68acf03\") " pod="openshift-monitoring/node-exporter-7j5j6"
Oct 30 21:50:05.393497 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:05.393440    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f938a93b-29dd-4e7d-b8e9-444df68acf03-node-exporter-accelerators-collector-config\") pod \"node-exporter-7j5j6\" (UID: \"f938a93b-29dd-4e7d-b8e9-444df68acf03\") " pod="openshift-monitoring/node-exporter-7j5j6"
Oct 30 21:50:05.393668 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:05.393649    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f938a93b-29dd-4e7d-b8e9-444df68acf03-metrics-client-ca\") pod \"node-exporter-7j5j6\" (UID: \"f938a93b-29dd-4e7d-b8e9-444df68acf03\") " pod="openshift-monitoring/node-exporter-7j5j6"
Oct 30 21:50:05.394980 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:05.394962    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f938a93b-29dd-4e7d-b8e9-444df68acf03-node-exporter-tls\") pod \"node-exporter-7j5j6\" (UID: \"f938a93b-29dd-4e7d-b8e9-444df68acf03\") " pod="openshift-monitoring/node-exporter-7j5j6"
Oct 30 21:50:05.395102 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:05.395084    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f938a93b-29dd-4e7d-b8e9-444df68acf03-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7j5j6\" (UID: \"f938a93b-29dd-4e7d-b8e9-444df68acf03\") " pod="openshift-monitoring/node-exporter-7j5j6"
Oct 30 21:50:05.451928 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:05.451866    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jt5t\" (UniqueName: \"kubernetes.io/projected/f938a93b-29dd-4e7d-b8e9-444df68acf03-kube-api-access-6jt5t\") pod \"node-exporter-7j5j6\" (UID: \"f938a93b-29dd-4e7d-b8e9-444df68acf03\") " pod="openshift-monitoring/node-exporter-7j5j6"
Oct 30 21:50:05.459791 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:05.459770    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pvl2l"
Oct 30 21:50:05.506635 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:05.506539    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7j5j6"
Oct 30 21:50:05.517071 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:50:05.517043    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf938a93b_29dd_4e7d_b8e9_444df68acf03.slice/crio-b80c4122abd5858b56886f4d518ab4c81848d68f3a7fd253c2ebc4930ac56b10 WatchSource:0}: Error finding container b80c4122abd5858b56886f4d518ab4c81848d68f3a7fd253c2ebc4930ac56b10: Status 404 returned error can't find the container with id b80c4122abd5858b56886f4d518ab4c81848d68f3a7fd253c2ebc4930ac56b10
Oct 30 21:50:05.551553 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:05.551529    2573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-69874d4cdd-zhlp5"
Oct 30 21:50:05.551629 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:05.551581    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-69874d4cdd-zhlp5"
Oct 30 21:50:05.555996 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:05.555979    2573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-69874d4cdd-zhlp5"
Oct 30 21:50:05.760339 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:05.760244    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7j5j6" event={"ID":"f938a93b-29dd-4e7d-b8e9-444df68acf03","Type":"ContainerStarted","Data":"b80c4122abd5858b56886f4d518ab4c81848d68f3a7fd253c2ebc4930ac56b10"}
Oct 30 21:50:05.764118 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:05.764093    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-69874d4cdd-zhlp5"
Oct 30 21:50:05.875478 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:05.875451    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-pvl2l"]
Oct 30 21:50:05.877657 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:50:05.877633    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddec1cd54_fec6_4c12_9d37_a1b0b1a7ffc0.slice/crio-99e016196628ee432af4e398621a5760c9b2739cc2b5a123742aa8523e334b05 WatchSource:0}: Error finding container 99e016196628ee432af4e398621a5760c9b2739cc2b5a123742aa8523e334b05: Status 404 returned error can't find the container with id 99e016196628ee432af4e398621a5760c9b2739cc2b5a123742aa8523e334b05
Oct 30 21:50:06.588668 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:06.588638    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf"
Oct 30 21:50:06.688901 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:06.688831    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-gmbbw_82f75dd5-40cd-4cb3-9111-6213d6b43679/node-ca/0.log"
Oct 30 21:50:06.765549 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:06.765513    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pvl2l" event={"ID":"dec1cd54-fec6-4c12-9d37-a1b0b1a7ffc0","Type":"ContainerStarted","Data":"99e016196628ee432af4e398621a5760c9b2739cc2b5a123742aa8523e334b05"}
Oct 30 21:50:07.270223 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:07.270193    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-hr227_516573f9-4ada-4868-b885-f61b9640e6bf/serve-healthcheck-canary/0.log"
Oct 30 21:50:07.365217 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:07.365177    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 21:50:07.368383 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:07.368354    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:50:07.412178 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:07.412135    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5138fe8c-9927-4d05-8fe5-5556072ed595-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5138fe8c-9927-4d05-8fe5-5556072ed595\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:50:07.412353 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:07.412184    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5138fe8c-9927-4d05-8fe5-5556072ed595-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5138fe8c-9927-4d05-8fe5-5556072ed595\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:50:07.412353 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:07.412334    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5138fe8c-9927-4d05-8fe5-5556072ed595-web-config\") pod \"alertmanager-main-0\" (UID: \"5138fe8c-9927-4d05-8fe5-5556072ed595\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:50:07.412472 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:07.412418    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5138fe8c-9927-4d05-8fe5-5556072ed595-config-out\") pod \"alertmanager-main-0\" (UID: \"5138fe8c-9927-4d05-8fe5-5556072ed595\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:50:07.412472 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:07.412447    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5138fe8c-9927-4d05-8fe5-5556072ed595-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5138fe8c-9927-4d05-8fe5-5556072ed595\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:50:07.412564 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:07.412474    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5138fe8c-9927-4d05-8fe5-5556072ed595-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5138fe8c-9927-4d05-8fe5-5556072ed595\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:50:07.412564 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:07.412523    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6v9rf\" (UniqueName: \"kubernetes.io/projected/5138fe8c-9927-4d05-8fe5-5556072ed595-kube-api-access-6v9rf\") pod \"alertmanager-main-0\" (UID: \"5138fe8c-9927-4d05-8fe5-5556072ed595\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:50:07.412658 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:07.412624    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5138fe8c-9927-4d05-8fe5-5556072ed595-config-volume\") pod \"alertmanager-main-0\" (UID: \"5138fe8c-9927-4d05-8fe5-5556072ed595\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:50:07.412710 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:07.412671    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5138fe8c-9927-4d05-8fe5-5556072ed595-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5138fe8c-9927-4d05-8fe5-5556072ed595\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:50:07.412767 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:07.412721    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5138fe8c-9927-4d05-8fe5-5556072ed595-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5138fe8c-9927-4d05-8fe5-5556072ed595\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:50:07.412843 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:07.412829    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5138fe8c-9927-4d05-8fe5-5556072ed595-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5138fe8c-9927-4d05-8fe5-5556072ed595\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:50:07.412902 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:07.412865    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5138fe8c-9927-4d05-8fe5-5556072ed595-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5138fe8c-9927-4d05-8fe5-5556072ed595\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:50:07.412902 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:07.412894    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5138fe8c-9927-4d05-8fe5-5556072ed595-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5138fe8c-9927-4d05-8fe5-5556072ed595\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:50:07.419428 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:07.419382    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Oct 30 21:50:07.419789 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:07.419656    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Oct 30 21:50:07.419789 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:07.419693    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Oct 30 21:50:07.419789 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:07.419744    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Oct 30 21:50:07.419974 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:07.419898    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-lxh58\""
Oct 30 21:50:07.514713 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:07.513502    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5138fe8c-9927-4d05-8fe5-5556072ed595-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5138fe8c-9927-4d05-8fe5-5556072ed595\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:50:07.514713 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:07.513551    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5138fe8c-9927-4d05-8fe5-5556072ed595-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5138fe8c-9927-4d05-8fe5-5556072ed595\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:50:07.514713 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:07.513598    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5138fe8c-9927-4d05-8fe5-5556072ed595-web-config\") pod \"alertmanager-main-0\" (UID: \"5138fe8c-9927-4d05-8fe5-5556072ed595\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:50:07.514713 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:07.513640    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5138fe8c-9927-4d05-8fe5-5556072ed595-config-out\") pod \"alertmanager-main-0\" (UID: \"5138fe8c-9927-4d05-8fe5-5556072ed595\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:50:07.514713 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:07.513664    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5138fe8c-9927-4d05-8fe5-5556072ed595-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5138fe8c-9927-4d05-8fe5-5556072ed595\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:50:07.514713 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:07.513690    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5138fe8c-9927-4d05-8fe5-5556072ed595-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5138fe8c-9927-4d05-8fe5-5556072ed595\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:50:07.514713 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:07.513721    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6v9rf\" (UniqueName: \"kubernetes.io/projected/5138fe8c-9927-4d05-8fe5-5556072ed595-kube-api-access-6v9rf\") pod \"alertmanager-main-0\" (UID: \"5138fe8c-9927-4d05-8fe5-5556072ed595\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:50:07.514713 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:07.513758    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5138fe8c-9927-4d05-8fe5-5556072ed595-config-volume\") pod \"alertmanager-main-0\" (UID: \"5138fe8c-9927-4d05-8fe5-5556072ed595\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:50:07.514713 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:07.513779    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5138fe8c-9927-4d05-8fe5-5556072ed595-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5138fe8c-9927-4d05-8fe5-5556072ed595\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:50:07.514713 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:07.513809    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5138fe8c-9927-4d05-8fe5-5556072ed595-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5138fe8c-9927-4d05-8fe5-5556072ed595\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:50:07.514713 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:07.513844    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5138fe8c-9927-4d05-8fe5-5556072ed595-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5138fe8c-9927-4d05-8fe5-5556072ed595\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:50:07.514713 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:07.513866    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5138fe8c-9927-4d05-8fe5-5556072ed595-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5138fe8c-9927-4d05-8fe5-5556072ed595\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:50:07.514713 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:07.513893    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5138fe8c-9927-4d05-8fe5-5556072ed595-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5138fe8c-9927-4d05-8fe5-5556072ed595\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:50:07.514713 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:07.513974    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Oct 30 21:50:07.514713 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:07.514069    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Oct 30 21:50:07.514713 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:07.514128    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Oct 30 21:50:07.514713 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:07.514245    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Oct 30 21:50:07.514713 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:07.514358    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Oct 30 21:50:07.516630 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:07.516602    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5138fe8c-9927-4d05-8fe5-5556072ed595-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5138fe8c-9927-4d05-8fe5-5556072ed595\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:50:07.517210 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:07.517181    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5138fe8c-9927-4d05-8fe5-5556072ed595-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5138fe8c-9927-4d05-8fe5-5556072ed595\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:50:07.517764 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:07.517413    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5138fe8c-9927-4d05-8fe5-5556072ed595-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5138fe8c-9927-4d05-8fe5-5556072ed595\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:50:07.517764 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:07.517570    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 21:50:07.517764 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:07.517722    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5138fe8c-9927-4d05-8fe5-5556072ed595-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5138fe8c-9927-4d05-8fe5-5556072ed595\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:50:07.518032 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:07.518012    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5138fe8c-9927-4d05-8fe5-5556072ed595-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5138fe8c-9927-4d05-8fe5-5556072ed595\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:50:07.518759 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:07.518740    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5138fe8c-9927-4d05-8fe5-5556072ed595-web-config\") pod \"alertmanager-main-0\" (UID: \"5138fe8c-9927-4d05-8fe5-5556072ed595\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:50:07.518881 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:07.518857    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5138fe8c-9927-4d05-8fe5-5556072ed595-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5138fe8c-9927-4d05-8fe5-5556072ed595\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:50:07.519882 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:07.519857    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5138fe8c-9927-4d05-8fe5-5556072ed595-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5138fe8c-9927-4d05-8fe5-5556072ed595\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:50:07.520139 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:07.520112    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5138fe8c-9927-4d05-8fe5-5556072ed595-config-volume\") pod \"alertmanager-main-0\" (UID: \"5138fe8c-9927-4d05-8fe5-5556072ed595\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:50:07.520343 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:07.520323    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5138fe8c-9927-4d05-8fe5-5556072ed595-config-out\") pod \"alertmanager-main-0\" (UID: \"5138fe8c-9927-4d05-8fe5-5556072ed595\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:50:07.520871 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:07.520847    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5138fe8c-9927-4d05-8fe5-5556072ed595-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5138fe8c-9927-4d05-8fe5-5556072ed595\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:50:07.521354 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:07.521338    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5138fe8c-9927-4d05-8fe5-5556072ed595-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5138fe8c-9927-4d05-8fe5-5556072ed595\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:50:07.573926 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:07.573897    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6v9rf\" (UniqueName: \"kubernetes.io/projected/5138fe8c-9927-4d05-8fe5-5556072ed595-kube-api-access-6v9rf\") pod \"alertmanager-main-0\" (UID: \"5138fe8c-9927-4d05-8fe5-5556072ed595\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:50:07.698211 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:07.698187    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:50:07.772863 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:07.771951    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pvl2l" event={"ID":"dec1cd54-fec6-4c12-9d37-a1b0b1a7ffc0","Type":"ContainerStarted","Data":"e3a1502114a90da5a1a5ec95f47bbedcabd156d04fed59d59756f32212d63687"}
Oct 30 21:50:07.772863 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:07.772029    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pvl2l" event={"ID":"dec1cd54-fec6-4c12-9d37-a1b0b1a7ffc0","Type":"ContainerStarted","Data":"ed8ac5bfe02beead1ce1766f14e57b82edb4604bc388cf3a65895ec8ed03ecb6"}
Oct 30 21:50:07.773512 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:07.773488    2573 generic.go:356] "Generic (PLEG): container finished" podID="f938a93b-29dd-4e7d-b8e9-444df68acf03" containerID="a2140afa979bca517055031d1bee6eddcc47ed7fcc16f3bba2d16f3da48c99d7" exitCode=0
Oct 30 21:50:07.773582 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:07.773525    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7j5j6" event={"ID":"f938a93b-29dd-4e7d-b8e9-444df68acf03","Type":"ContainerDied","Data":"a2140afa979bca517055031d1bee6eddcc47ed7fcc16f3bba2d16f3da48c99d7"}
Oct 30 21:50:08.124845 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:08.124808    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 21:50:08.130140 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:50:08.130114    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5138fe8c_9927_4d05_8fe5_5556072ed595.slice/crio-dc24a46b097bebb5444228144f254835785f13e6c1da310ba365dff351a254f6 WatchSource:0}: Error finding container dc24a46b097bebb5444228144f254835785f13e6c1da310ba365dff351a254f6: Status 404 returned error can't find the container with id dc24a46b097bebb5444228144f254835785f13e6c1da310ba365dff351a254f6
Oct 30 21:50:08.778288 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:08.778253    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7j5j6" event={"ID":"f938a93b-29dd-4e7d-b8e9-444df68acf03","Type":"ContainerStarted","Data":"cf2d36db9696f4251b92ba3b28b56adf6937310cca49dae365a85428ee86005c"}
Oct 30 21:50:08.778737 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:08.778315    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7j5j6" event={"ID":"f938a93b-29dd-4e7d-b8e9-444df68acf03","Type":"ContainerStarted","Data":"f851a09a43c6be3aa313c216a270a6b53c162b9216af2e9aee1929992a11808c"}
Oct 30 21:50:08.780271 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:08.780248    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pvl2l" event={"ID":"dec1cd54-fec6-4c12-9d37-a1b0b1a7ffc0","Type":"ContainerStarted","Data":"39b8dd1259fb48df5ba493b3b18ce4e9f1af83ea864603d7024cd87a55397ea5"}
Oct 30 21:50:08.781231 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:08.781213    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5138fe8c-9927-4d05-8fe5-5556072ed595","Type":"ContainerStarted","Data":"dc24a46b097bebb5444228144f254835785f13e6c1da310ba365dff351a254f6"}
Oct 30 21:50:08.903619 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:08.903568    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-7j5j6" podStartSLOduration=2.175695167 podStartE2EDuration="3.903553966s" totalImagesPullingTime="1.655311127s" totalInitContainerRuntime="72.547672ms" isStatefulPod=true podCreationTimestamp="2026-10-30 21:50:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:50:08.902465681 +0000 UTC m=+60.025090409" watchObservedRunningTime="2026-10-30 21:50:08.903553966 +0000 UTC m=+60.026178693"
Oct 30 21:50:09.018365 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:09.018309    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pvl2l" podStartSLOduration=2.287049027 podStartE2EDuration="4.018292236s" totalImagesPullingTime="1.731243209s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:50:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:50:09.017957455 +0000 UTC m=+60.140582183" watchObservedRunningTime="2026-10-30 21:50:09.018292236 +0000 UTC m=+60.140917104"
Oct 30 21:50:09.745972 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:09.745941    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-fdmdz"
Oct 30 21:50:10.163892 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:10.163859    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-dc7c7d47d-ghdn7"]
Oct 30 21:50:10.166282 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:10.166255    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-dc7c7d47d-ghdn7"
Oct 30 21:50:10.219015 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:10.218982    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Oct 30 21:50:10.219151 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:10.219136    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Oct 30 21:50:10.220409 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:10.220381    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Oct 30 21:50:10.220500 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:10.220486    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-gnnmn\""
Oct 30 21:50:10.220548 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:10.220499    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Oct 30 21:50:10.220548 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:10.220541    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-920fbr0654gnu\""
Oct 30 21:50:10.230431 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:10.230396    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-dc7c7d47d-ghdn7"]
Oct 30 21:50:10.236124 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:10.236103    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/a3459ebb-2fc2-4a82-b9c0-de5542d667e1-audit-log\") pod \"metrics-server-dc7c7d47d-ghdn7\" (UID: \"a3459ebb-2fc2-4a82-b9c0-de5542d667e1\") " pod="openshift-monitoring/metrics-server-dc7c7d47d-ghdn7"
Oct 30 21:50:10.236204 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:10.236139    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/a3459ebb-2fc2-4a82-b9c0-de5542d667e1-client-ca-bundle\") pod \"metrics-server-dc7c7d47d-ghdn7\" (UID: \"a3459ebb-2fc2-4a82-b9c0-de5542d667e1\") " pod="openshift-monitoring/metrics-server-dc7c7d47d-ghdn7"
Oct 30 21:50:10.236204 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:10.236160    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/a3459ebb-2fc2-4a82-b9c0-de5542d667e1-metrics-server-audit-profiles\") pod \"metrics-server-dc7c7d47d-ghdn7\" (UID: \"a3459ebb-2fc2-4a82-b9c0-de5542d667e1\") " pod="openshift-monitoring/metrics-server-dc7c7d47d-ghdn7"
Oct 30 21:50:10.236341 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:10.236289    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/a3459ebb-2fc2-4a82-b9c0-de5542d667e1-secret-metrics-server-tls\") pod \"metrics-server-dc7c7d47d-ghdn7\" (UID: \"a3459ebb-2fc2-4a82-b9c0-de5542d667e1\") " pod="openshift-monitoring/metrics-server-dc7c7d47d-ghdn7"
Oct 30 21:50:10.236341 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:10.236319    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3459ebb-2fc2-4a82-b9c0-de5542d667e1-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-dc7c7d47d-ghdn7\" (UID: \"a3459ebb-2fc2-4a82-b9c0-de5542d667e1\") " pod="openshift-monitoring/metrics-server-dc7c7d47d-ghdn7"
Oct 30 21:50:10.236341 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:10.236336    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/a3459ebb-2fc2-4a82-b9c0-de5542d667e1-secret-metrics-server-client-certs\") pod \"metrics-server-dc7c7d47d-ghdn7\" (UID: \"a3459ebb-2fc2-4a82-b9c0-de5542d667e1\") " pod="openshift-monitoring/metrics-server-dc7c7d47d-ghdn7"
Oct 30 21:50:10.236508 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:10.236491    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hdbq\" (UniqueName: \"kubernetes.io/projected/a3459ebb-2fc2-4a82-b9c0-de5542d667e1-kube-api-access-6hdbq\") pod \"metrics-server-dc7c7d47d-ghdn7\" (UID: \"a3459ebb-2fc2-4a82-b9c0-de5542d667e1\") " pod="openshift-monitoring/metrics-server-dc7c7d47d-ghdn7"
Oct 30 21:50:10.337378 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:10.337344    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/a3459ebb-2fc2-4a82-b9c0-de5542d667e1-secret-metrics-server-tls\") pod \"metrics-server-dc7c7d47d-ghdn7\" (UID: \"a3459ebb-2fc2-4a82-b9c0-de5542d667e1\") " pod="openshift-monitoring/metrics-server-dc7c7d47d-ghdn7"
Oct 30 21:50:10.337378 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:10.337381    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3459ebb-2fc2-4a82-b9c0-de5542d667e1-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-dc7c7d47d-ghdn7\" (UID: \"a3459ebb-2fc2-4a82-b9c0-de5542d667e1\") " pod="openshift-monitoring/metrics-server-dc7c7d47d-ghdn7"
Oct 30 21:50:10.337622 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:10.337421    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/a3459ebb-2fc2-4a82-b9c0-de5542d667e1-secret-metrics-server-client-certs\") pod \"metrics-server-dc7c7d47d-ghdn7\" (UID: \"a3459ebb-2fc2-4a82-b9c0-de5542d667e1\") " pod="openshift-monitoring/metrics-server-dc7c7d47d-ghdn7"
Oct 30 21:50:10.337622 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:10.337475    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hdbq\" (UniqueName: \"kubernetes.io/projected/a3459ebb-2fc2-4a82-b9c0-de5542d667e1-kube-api-access-6hdbq\") pod \"metrics-server-dc7c7d47d-ghdn7\" (UID: \"a3459ebb-2fc2-4a82-b9c0-de5542d667e1\") " pod="openshift-monitoring/metrics-server-dc7c7d47d-ghdn7"
Oct 30 21:50:10.337622 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:10.337495    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/a3459ebb-2fc2-4a82-b9c0-de5542d667e1-audit-log\") pod \"metrics-server-dc7c7d47d-ghdn7\" (UID: \"a3459ebb-2fc2-4a82-b9c0-de5542d667e1\") " pod="openshift-monitoring/metrics-server-dc7c7d47d-ghdn7"
Oct 30 21:50:10.337622 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:10.337554    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/a3459ebb-2fc2-4a82-b9c0-de5542d667e1-client-ca-bundle\") pod \"metrics-server-dc7c7d47d-ghdn7\" (UID: \"a3459ebb-2fc2-4a82-b9c0-de5542d667e1\") " pod="openshift-monitoring/metrics-server-dc7c7d47d-ghdn7"
Oct 30 21:50:10.337622 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:10.337590    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/a3459ebb-2fc2-4a82-b9c0-de5542d667e1-metrics-server-audit-profiles\") pod \"metrics-server-dc7c7d47d-ghdn7\" (UID: \"a3459ebb-2fc2-4a82-b9c0-de5542d667e1\") " pod="openshift-monitoring/metrics-server-dc7c7d47d-ghdn7"
Oct 30 21:50:10.337971 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:10.337947    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/a3459ebb-2fc2-4a82-b9c0-de5542d667e1-audit-log\") pod \"metrics-server-dc7c7d47d-ghdn7\" (UID: \"a3459ebb-2fc2-4a82-b9c0-de5542d667e1\") " pod="openshift-monitoring/metrics-server-dc7c7d47d-ghdn7"
Oct 30 21:50:10.338205 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:10.338185    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3459ebb-2fc2-4a82-b9c0-de5542d667e1-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-dc7c7d47d-ghdn7\" (UID: \"a3459ebb-2fc2-4a82-b9c0-de5542d667e1\") " pod="openshift-monitoring/metrics-server-dc7c7d47d-ghdn7"
Oct 30 21:50:10.338803 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:10.338773    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/a3459ebb-2fc2-4a82-b9c0-de5542d667e1-metrics-server-audit-profiles\") pod \"metrics-server-dc7c7d47d-ghdn7\" (UID: \"a3459ebb-2fc2-4a82-b9c0-de5542d667e1\") " pod="openshift-monitoring/metrics-server-dc7c7d47d-ghdn7"
Oct 30 21:50:10.340126 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:10.340101    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/a3459ebb-2fc2-4a82-b9c0-de5542d667e1-secret-metrics-server-client-certs\") pod \"metrics-server-dc7c7d47d-ghdn7\" (UID: \"a3459ebb-2fc2-4a82-b9c0-de5542d667e1\") " pod="openshift-monitoring/metrics-server-dc7c7d47d-ghdn7"
Oct 30 21:50:10.340231 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:10.340151    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/a3459ebb-2fc2-4a82-b9c0-de5542d667e1-secret-metrics-server-tls\") pod \"metrics-server-dc7c7d47d-ghdn7\" (UID: \"a3459ebb-2fc2-4a82-b9c0-de5542d667e1\") " pod="openshift-monitoring/metrics-server-dc7c7d47d-ghdn7"
Oct 30 21:50:10.340291 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:10.340247    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/a3459ebb-2fc2-4a82-b9c0-de5542d667e1-client-ca-bundle\") pod \"metrics-server-dc7c7d47d-ghdn7\" (UID: \"a3459ebb-2fc2-4a82-b9c0-de5542d667e1\") " pod="openshift-monitoring/metrics-server-dc7c7d47d-ghdn7"
Oct 30 21:50:10.397319 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:10.397294    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hdbq\" (UniqueName: \"kubernetes.io/projected/a3459ebb-2fc2-4a82-b9c0-de5542d667e1-kube-api-access-6hdbq\") pod \"metrics-server-dc7c7d47d-ghdn7\" (UID: \"a3459ebb-2fc2-4a82-b9c0-de5542d667e1\") " pod="openshift-monitoring/metrics-server-dc7c7d47d-ghdn7"
Oct 30 21:50:10.477294 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:10.477218    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-dc7c7d47d-ghdn7"
Oct 30 21:50:10.789924 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:10.789845    2573 generic.go:356] "Generic (PLEG): container finished" podID="5138fe8c-9927-4d05-8fe5-5556072ed595" containerID="f7f271fecf3370b4e1a7117367120bc8e1a3ba6129d0a2171e8c002bee61190a" exitCode=0
Oct 30 21:50:10.789924 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:10.789895    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5138fe8c-9927-4d05-8fe5-5556072ed595","Type":"ContainerDied","Data":"f7f271fecf3370b4e1a7117367120bc8e1a3ba6129d0a2171e8c002bee61190a"}
Oct 30 21:50:10.888517 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:10.888489    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-dc7c7d47d-ghdn7"]
Oct 30 21:50:10.891663 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:50:10.891625    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3459ebb_2fc2_4a82_b9c0_de5542d667e1.slice/crio-a7fc187ce460fbaeec60b28f3c313837ba07a8341b8639946b2382e42dde1919 WatchSource:0}: Error finding container a7fc187ce460fbaeec60b28f3c313837ba07a8341b8639946b2382e42dde1919: Status 404 returned error can't find the container with id a7fc187ce460fbaeec60b28f3c313837ba07a8341b8639946b2382e42dde1919
Oct 30 21:50:11.793726 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:11.793690    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-dc7c7d47d-ghdn7" event={"ID":"a3459ebb-2fc2-4a82-b9c0-de5542d667e1","Type":"ContainerStarted","Data":"a7fc187ce460fbaeec60b28f3c313837ba07a8341b8639946b2382e42dde1919"}
Oct 30 21:50:13.804128 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:13.804094    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5138fe8c-9927-4d05-8fe5-5556072ed595","Type":"ContainerStarted","Data":"d2766685272356bcd2c86bf5bf3b1cce9d467808eba84132b88fbbe31d1643ef"}
Oct 30 21:50:13.804587 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:13.804162    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5138fe8c-9927-4d05-8fe5-5556072ed595","Type":"ContainerStarted","Data":"4226791aeb98d543049bf57efc9d5e8b19e4cf53528ff2a742fda907deb4c396"}
Oct 30 21:50:13.804587 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:13.804201    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5138fe8c-9927-4d05-8fe5-5556072ed595","Type":"ContainerStarted","Data":"a93d15733e8cc09f6a51bfb22088505a1330edba1d3d0d6106aea54679c62b22"}
Oct 30 21:50:13.804587 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:13.804229    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5138fe8c-9927-4d05-8fe5-5556072ed595","Type":"ContainerStarted","Data":"a4213395543a6ef339728f8523f33f15faf75bdee21587970d4f87f953cf526e"}
Oct 30 21:50:13.804587 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:13.804252    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5138fe8c-9927-4d05-8fe5-5556072ed595","Type":"ContainerStarted","Data":"ebbf6d346fb1ef77b512d07e571efc8794da78e6d86fc712a611858d3530ec5d"}
Oct 30 21:50:13.805684 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:13.805659    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-dc7c7d47d-ghdn7" event={"ID":"a3459ebb-2fc2-4a82-b9c0-de5542d667e1","Type":"ContainerStarted","Data":"ca3c6af5f954c0d3e7da2e177aaf92c81f06d89529a0ef2effafc877c7e3593f"}
Oct 30 21:50:13.958887 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:13.958810    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-dc7c7d47d-ghdn7" podStartSLOduration=1.839292994 podStartE2EDuration="3.958796323s" totalImagesPullingTime="2.119503329s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:50:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:50:13.957820985 +0000 UTC m=+65.080445724" watchObservedRunningTime="2026-10-30 21:50:13.958796323 +0000 UTC m=+65.081421051"
Oct 30 21:50:14.637980 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:14.637955    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 21:50:14.641782 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:14.641762    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:14.675941 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:14.675900    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:14.676056 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:14.675952    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:14.676056 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:14.676026    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:14.676130 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:14.676082    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:14.676167 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:14.676131    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:14.676167 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:14.676151    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:14.676225 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:14.676169    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:14.676256 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:14.676241    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cr8x2\" (UniqueName: \"kubernetes.io/projected/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-kube-api-access-cr8x2\") pod \"prometheus-k8s-0\" (UID: \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:14.676285 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:14.676266    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-config\") pod \"prometheus-k8s-0\" (UID: \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:14.676325 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:14.676282    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-config-out\") pod \"prometheus-k8s-0\" (UID: \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:14.676325 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:14.676318    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:14.676420 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:14.676345    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:14.676420 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:14.676367    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:14.676494 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:14.676464    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-web-config\") pod \"prometheus-k8s-0\" (UID: \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:14.676494 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:14.676484    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:14.676591 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:14.676507    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:14.676643 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:14.676583    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:14.676643 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:14.676623    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:14.676740 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:14.676659    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:14.676807 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:14.676790    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:14.691372 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:14.691173    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 21:50:14.736319 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:14.736293    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Oct 30 21:50:14.736466 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:14.736341    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Oct 30 21:50:14.736579 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:14.736563    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Oct 30 21:50:14.736579 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:14.736572    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Oct 30 21:50:14.736715 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:14.736636    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-kfpck\""
Oct 30 21:50:14.736715 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:14.736649    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Oct 30 21:50:14.736715 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:14.736656    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-bmu48km85ucup\""
Oct 30 21:50:14.736829 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:14.736773    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Oct 30 21:50:14.736829 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:14.736788    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Oct 30 21:50:14.736829 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:14.736778    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Oct 30 21:50:14.736829 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:14.736813    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Oct 30 21:50:14.777668 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:14.777638    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-web-config\") pod \"prometheus-k8s-0\" (UID: \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:14.777668 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:14.777673    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:14.777891 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:14.777691    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:14.777891 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:14.777717    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:14.777891 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:14.777742    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:14.777891 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:14.777771    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:14.777891 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:14.777833    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:14.777891 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:14.777874    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:14.778177 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:14.777901    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:14.778177 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:14.777933    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:14.778177 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:14.777955    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:14.778177 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:14.777984    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:14.778177 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:14.778010    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:14.778177 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:14.778031    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:14.778177 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:14.778067    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cr8x2\" (UniqueName: \"kubernetes.io/projected/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-kube-api-access-cr8x2\") pod \"prometheus-k8s-0\" (UID: \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:14.778177 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:14.778092    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-config\") pod \"prometheus-k8s-0\" (UID: \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:14.778177 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:14.778115    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-config-out\") pod \"prometheus-k8s-0\" (UID: \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:14.778177 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:14.778139    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:14.778177 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:14.778166    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:14.779878 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:14.778191    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:14.779878 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:14.778666    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:14.779878 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:14.779171    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:14.779878 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:14.779469    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:14.781346 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:14.781319    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-web-config\") pod \"prometheus-k8s-0\" (UID: \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:14.781474 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:14.781433    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:14.781565 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:14.781533    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:14.781565 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:14.781547    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:14.781747 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:14.781569    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:14.782078 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:14.782051    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:14.783079 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:14.783047    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:14.783163 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:14.783080    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:14.783163 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:14.783098    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:14.783232 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:14.783158    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-config\") pod \"prometheus-k8s-0\" (UID: \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:14.783591 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:14.783571    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-config-out\") pod \"prometheus-k8s-0\" (UID: \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:14.783880 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:14.783864    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Oct 30 21:50:14.783880 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:14.783867    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Oct 30 21:50:14.784000 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:14.783892    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Oct 30 21:50:14.788999 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:14.788981    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:14.789341 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:14.789325    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:14.789678 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:14.789658    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:14.812450 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:14.812421    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5138fe8c-9927-4d05-8fe5-5556072ed595","Type":"ContainerStarted","Data":"0026372bec6ebf4cc2f80b456ae15eafc47d5675ef85d041318db9169cdec013"}
Oct 30 21:50:14.832864 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:14.832839    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Oct 30 21:50:14.841127 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:14.841106    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:14.879346 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:14.878761    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Oct 30 21:50:14.880131 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:14.880108    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:14.881555 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:14.881536    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cr8x2\" (UniqueName: \"kubernetes.io/projected/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-kube-api-access-cr8x2\") pod \"prometheus-k8s-0\" (UID: \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:14.961895 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:14.961867    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:15.014686 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:15.014631    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.901935178 podStartE2EDuration="8.014611812s" totalImagesPullingTime="5.098437129s" totalInitContainerRuntime="14.239505ms" isStatefulPod=false podCreationTimestamp="2026-10-30 21:50:07 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:50:15.010659062 +0000 UTC m=+66.133283800" watchObservedRunningTime="2026-10-30 21:50:15.014611812 +0000 UTC m=+66.137236541"
Oct 30 21:50:15.282410 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:15.282316    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6980f40-89cf-4f3d-8cd5-91969ae5a276-metrics-certs\") pod \"network-metrics-daemon-xbh9m\" (UID: \"a6980f40-89cf-4f3d-8cd5-91969ae5a276\") " pod="openshift-multus/network-metrics-daemon-xbh9m"
Oct 30 21:50:15.282410 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:15.282375    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b954bab2-9dd7-4b39-bda6-3fec87381aff-original-pull-secret\") pod \"global-pull-secret-syncer-cck7d\" (UID: \"b954bab2-9dd7-4b39-bda6-3fec87381aff\") " pod="kube-system/global-pull-secret-syncer-cck7d"
Oct 30 21:50:15.331158 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:15.331127    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 21:50:15.331278 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:15.331224    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 21:50:15.335465 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:15.335435    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b954bab2-9dd7-4b39-bda6-3fec87381aff-original-pull-secret\") pod \"global-pull-secret-syncer-cck7d\" (UID: \"b954bab2-9dd7-4b39-bda6-3fec87381aff\") " pod="kube-system/global-pull-secret-syncer-cck7d"
Oct 30 21:50:15.335606 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:15.335474    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6980f40-89cf-4f3d-8cd5-91969ae5a276-metrics-certs\") pod \"network-metrics-daemon-xbh9m\" (UID: \"a6980f40-89cf-4f3d-8cd5-91969ae5a276\") " pod="openshift-multus/network-metrics-daemon-xbh9m"
Oct 30 21:50:15.384019 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:15.383993    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6x28\" (UniqueName: \"kubernetes.io/projected/55619a2e-f6e9-4719-b362-30b69d553967-kube-api-access-j6x28\") pod \"network-check-target-w9tx8\" (UID: \"55619a2e-f6e9-4719-b362-30b69d553967\") " pod="openshift-network-diagnostics/network-check-target-w9tx8"
Oct 30 21:50:15.385634 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:15.385615    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 21:50:15.388446 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:50:15.388420    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef7ae922_6da7_4817_ba91_35e67b6ac8c5.slice/crio-266940a224ea0a54c1a8a2d9b4e5b33f7b149ab8f92cc7b373d4259ede742beb WatchSource:0}: Error finding container 266940a224ea0a54c1a8a2d9b4e5b33f7b149ab8f92cc7b373d4259ede742beb: Status 404 returned error can't find the container with id 266940a224ea0a54c1a8a2d9b4e5b33f7b149ab8f92cc7b373d4259ede742beb
Oct 30 21:50:15.432954 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:15.432930    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 21:50:15.440384 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:15.440367    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-jbflf\""
Oct 30 21:50:15.441447 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:15.441433    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cck7d"
Oct 30 21:50:15.483581 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:15.483558    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 21:50:15.492938 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:15.491305    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6x28\" (UniqueName: \"kubernetes.io/projected/55619a2e-f6e9-4719-b362-30b69d553967-kube-api-access-j6x28\") pod \"network-check-target-w9tx8\" (UID: \"55619a2e-f6e9-4719-b362-30b69d553967\") " pod="openshift-network-diagnostics/network-check-target-w9tx8"
Oct 30 21:50:15.644851 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:15.644819    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9bjlv\""
Oct 30 21:50:15.654081 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:15.654057    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xbh9m"
Oct 30 21:50:15.745275 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:15.745242    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-9mgc8\""
Oct 30 21:50:15.747351 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:15.747333    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w9tx8"
Oct 30 21:50:15.807361 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:15.807329    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-cck7d"]
Oct 30 21:50:15.810762 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:50:15.810739    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb954bab2_9dd7_4b39_bda6_3fec87381aff.slice/crio-2d4d5b332a32797e444f7b86043649ef9e8cd85554203c183f4263d812176bb9 WatchSource:0}: Error finding container 2d4d5b332a32797e444f7b86043649ef9e8cd85554203c183f4263d812176bb9: Status 404 returned error can't find the container with id 2d4d5b332a32797e444f7b86043649ef9e8cd85554203c183f4263d812176bb9
Oct 30 21:50:15.816915 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:15.816888    2573 generic.go:356] "Generic (PLEG): container finished" podID="ef7ae922-6da7-4817-ba91-35e67b6ac8c5" containerID="258fd66deee3797d90e92c40203fe9ebf6228e43b399cf1d0332b8034deed533" exitCode=0
Oct 30 21:50:15.817217 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:15.816970    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ef7ae922-6da7-4817-ba91-35e67b6ac8c5","Type":"ContainerDied","Data":"258fd66deee3797d90e92c40203fe9ebf6228e43b399cf1d0332b8034deed533"}
Oct 30 21:50:15.817217 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:15.817036    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ef7ae922-6da7-4817-ba91-35e67b6ac8c5","Type":"ContainerStarted","Data":"266940a224ea0a54c1a8a2d9b4e5b33f7b149ab8f92cc7b373d4259ede742beb"}
Oct 30 21:50:15.818379 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:15.818347    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-cck7d" event={"ID":"b954bab2-9dd7-4b39-bda6-3fec87381aff","Type":"ContainerStarted","Data":"2d4d5b332a32797e444f7b86043649ef9e8cd85554203c183f4263d812176bb9"}
Oct 30 21:50:16.025142 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:16.025119    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xbh9m"]
Oct 30 21:50:16.027494 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:50:16.027452    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6980f40_89cf_4f3d_8cd5_91969ae5a276.slice/crio-54feac0378fd0894bff57a7d55fef92cb31ed8f5b2fcfe974fa7dcb286576b70 WatchSource:0}: Error finding container 54feac0378fd0894bff57a7d55fef92cb31ed8f5b2fcfe974fa7dcb286576b70: Status 404 returned error can't find the container with id 54feac0378fd0894bff57a7d55fef92cb31ed8f5b2fcfe974fa7dcb286576b70
Oct 30 21:50:16.111718 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:16.111685    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-w9tx8"]
Oct 30 21:50:16.114480 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:50:16.114451    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55619a2e_f6e9_4719_b362_30b69d553967.slice/crio-d96925669f74a34a6a8e0245a59131261ec4d0a7c88335b9eced7a1f596f1b04 WatchSource:0}: Error finding container d96925669f74a34a6a8e0245a59131261ec4d0a7c88335b9eced7a1f596f1b04: Status 404 returned error can't find the container with id d96925669f74a34a6a8e0245a59131261ec4d0a7c88335b9eced7a1f596f1b04
Oct 30 21:50:16.823471 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:16.823418    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-w9tx8" event={"ID":"55619a2e-f6e9-4719-b362-30b69d553967","Type":"ContainerStarted","Data":"d96925669f74a34a6a8e0245a59131261ec4d0a7c88335b9eced7a1f596f1b04"}
Oct 30 21:50:16.824627 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:16.824606    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xbh9m" event={"ID":"a6980f40-89cf-4f3d-8cd5-91969ae5a276","Type":"ContainerStarted","Data":"54feac0378fd0894bff57a7d55fef92cb31ed8f5b2fcfe974fa7dcb286576b70"}
Oct 30 21:50:18.837374 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:18.837336    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xbh9m" event={"ID":"a6980f40-89cf-4f3d-8cd5-91969ae5a276","Type":"ContainerStarted","Data":"f0d49a770935b5935a47a6a541e21ff58a1c6f2fdd83bab53663c0360dc57e84"}
Oct 30 21:50:18.837934 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:18.837428    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xbh9m" event={"ID":"a6980f40-89cf-4f3d-8cd5-91969ae5a276","Type":"ContainerStarted","Data":"b06422e7040d40ce7301f18ae0b947558df7af274290a6d9a971e3bc60e0e7a0"}
Oct 30 21:50:19.734339 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:19.734242    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-xbh9m" podStartSLOduration=67.810501354 podStartE2EDuration="1m9.734223161s" totalImagesPullingTime="1.923721807s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:49:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:50:18.951597956 +0000 UTC m=+70.074222695" watchObservedRunningTime="2026-10-30 21:50:19.734223161 +0000 UTC m=+70.856847890"
Oct 30 21:50:19.735863 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:19.735836    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-7cbdbcf4b5-rmbsz"]
Oct 30 21:50:19.740507 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:19.740486    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7cbdbcf4b5-rmbsz"
Oct 30 21:50:19.825281 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:19.825249    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fdf54bb9-f735-4810-82ee-d803099a1bcf-console-serving-cert\") pod \"console-7cbdbcf4b5-rmbsz\" (UID: \"fdf54bb9-f735-4810-82ee-d803099a1bcf\") " pod="openshift-console/console-7cbdbcf4b5-rmbsz"
Oct 30 21:50:19.825465 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:19.825292    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fdf54bb9-f735-4810-82ee-d803099a1bcf-trusted-ca-bundle\") pod \"console-7cbdbcf4b5-rmbsz\" (UID: \"fdf54bb9-f735-4810-82ee-d803099a1bcf\") " pod="openshift-console/console-7cbdbcf4b5-rmbsz"
Oct 30 21:50:19.825465 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:19.825319    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fdf54bb9-f735-4810-82ee-d803099a1bcf-service-ca\") pod \"console-7cbdbcf4b5-rmbsz\" (UID: \"fdf54bb9-f735-4810-82ee-d803099a1bcf\") " pod="openshift-console/console-7cbdbcf4b5-rmbsz"
Oct 30 21:50:19.825586 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:19.825459    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fdf54bb9-f735-4810-82ee-d803099a1bcf-tmp\") pod \"console-7cbdbcf4b5-rmbsz\" (UID: \"fdf54bb9-f735-4810-82ee-d803099a1bcf\") " pod="openshift-console/console-7cbdbcf4b5-rmbsz"
Oct 30 21:50:19.825586 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:19.825516    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fdf54bb9-f735-4810-82ee-d803099a1bcf-oauth-serving-cert\") pod \"console-7cbdbcf4b5-rmbsz\" (UID: \"fdf54bb9-f735-4810-82ee-d803099a1bcf\") " pod="openshift-console/console-7cbdbcf4b5-rmbsz"
Oct 30 21:50:19.825586 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:19.825579    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fdf54bb9-f735-4810-82ee-d803099a1bcf-console-config\") pod \"console-7cbdbcf4b5-rmbsz\" (UID: \"fdf54bb9-f735-4810-82ee-d803099a1bcf\") " pod="openshift-console/console-7cbdbcf4b5-rmbsz"
Oct 30 21:50:19.825690 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:19.825605    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5jdv\" (UniqueName: \"kubernetes.io/projected/fdf54bb9-f735-4810-82ee-d803099a1bcf-kube-api-access-s5jdv\") pod \"console-7cbdbcf4b5-rmbsz\" (UID: \"fdf54bb9-f735-4810-82ee-d803099a1bcf\") " pod="openshift-console/console-7cbdbcf4b5-rmbsz"
Oct 30 21:50:19.825690 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:19.825649    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fdf54bb9-f735-4810-82ee-d803099a1bcf-console-oauth-config\") pod \"console-7cbdbcf4b5-rmbsz\" (UID: \"fdf54bb9-f735-4810-82ee-d803099a1bcf\") " pod="openshift-console/console-7cbdbcf4b5-rmbsz"
Oct 30 21:50:19.926316 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:19.926274    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fdf54bb9-f735-4810-82ee-d803099a1bcf-console-serving-cert\") pod \"console-7cbdbcf4b5-rmbsz\" (UID: \"fdf54bb9-f735-4810-82ee-d803099a1bcf\") " pod="openshift-console/console-7cbdbcf4b5-rmbsz"
Oct 30 21:50:19.926761 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:19.926326    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fdf54bb9-f735-4810-82ee-d803099a1bcf-trusted-ca-bundle\") pod \"console-7cbdbcf4b5-rmbsz\" (UID: \"fdf54bb9-f735-4810-82ee-d803099a1bcf\") " pod="openshift-console/console-7cbdbcf4b5-rmbsz"
Oct 30 21:50:19.926761 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:19.926357    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fdf54bb9-f735-4810-82ee-d803099a1bcf-service-ca\") pod \"console-7cbdbcf4b5-rmbsz\" (UID: \"fdf54bb9-f735-4810-82ee-d803099a1bcf\") " pod="openshift-console/console-7cbdbcf4b5-rmbsz"
Oct 30 21:50:19.926761 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:19.926396    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fdf54bb9-f735-4810-82ee-d803099a1bcf-tmp\") pod \"console-7cbdbcf4b5-rmbsz\" (UID: \"fdf54bb9-f735-4810-82ee-d803099a1bcf\") " pod="openshift-console/console-7cbdbcf4b5-rmbsz"
Oct 30 21:50:19.926761 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:19.926470    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fdf54bb9-f735-4810-82ee-d803099a1bcf-oauth-serving-cert\") pod \"console-7cbdbcf4b5-rmbsz\" (UID: \"fdf54bb9-f735-4810-82ee-d803099a1bcf\") " pod="openshift-console/console-7cbdbcf4b5-rmbsz"
Oct 30 21:50:19.926761 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:19.926542    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fdf54bb9-f735-4810-82ee-d803099a1bcf-console-config\") pod \"console-7cbdbcf4b5-rmbsz\" (UID: \"fdf54bb9-f735-4810-82ee-d803099a1bcf\") " pod="openshift-console/console-7cbdbcf4b5-rmbsz"
Oct 30 21:50:19.926761 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:19.926567    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5jdv\" (UniqueName: \"kubernetes.io/projected/fdf54bb9-f735-4810-82ee-d803099a1bcf-kube-api-access-s5jdv\") pod \"console-7cbdbcf4b5-rmbsz\" (UID: \"fdf54bb9-f735-4810-82ee-d803099a1bcf\") " pod="openshift-console/console-7cbdbcf4b5-rmbsz"
Oct 30 21:50:19.926761 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:19.926611    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fdf54bb9-f735-4810-82ee-d803099a1bcf-console-oauth-config\") pod \"console-7cbdbcf4b5-rmbsz\" (UID: \"fdf54bb9-f735-4810-82ee-d803099a1bcf\") " pod="openshift-console/console-7cbdbcf4b5-rmbsz"
Oct 30 21:50:19.927177 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:19.927155    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fdf54bb9-f735-4810-82ee-d803099a1bcf-tmp\") pod \"console-7cbdbcf4b5-rmbsz\" (UID: \"fdf54bb9-f735-4810-82ee-d803099a1bcf\") " pod="openshift-console/console-7cbdbcf4b5-rmbsz"
Oct 30 21:50:19.927742 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:19.927716    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fdf54bb9-f735-4810-82ee-d803099a1bcf-console-config\") pod \"console-7cbdbcf4b5-rmbsz\" (UID: \"fdf54bb9-f735-4810-82ee-d803099a1bcf\") " pod="openshift-console/console-7cbdbcf4b5-rmbsz"
Oct 30 21:50:19.927842 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:19.927763    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fdf54bb9-f735-4810-82ee-d803099a1bcf-oauth-serving-cert\") pod \"console-7cbdbcf4b5-rmbsz\" (UID: \"fdf54bb9-f735-4810-82ee-d803099a1bcf\") " pod="openshift-console/console-7cbdbcf4b5-rmbsz"
Oct 30 21:50:19.927842 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:19.927760    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fdf54bb9-f735-4810-82ee-d803099a1bcf-service-ca\") pod \"console-7cbdbcf4b5-rmbsz\" (UID: \"fdf54bb9-f735-4810-82ee-d803099a1bcf\") " pod="openshift-console/console-7cbdbcf4b5-rmbsz"
Oct 30 21:50:19.930654 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:19.930510    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fdf54bb9-f735-4810-82ee-d803099a1bcf-console-oauth-config\") pod \"console-7cbdbcf4b5-rmbsz\" (UID: \"fdf54bb9-f735-4810-82ee-d803099a1bcf\") " pod="openshift-console/console-7cbdbcf4b5-rmbsz"
Oct 30 21:50:19.930654 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:19.930615    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fdf54bb9-f735-4810-82ee-d803099a1bcf-console-serving-cert\") pod \"console-7cbdbcf4b5-rmbsz\" (UID: \"fdf54bb9-f735-4810-82ee-d803099a1bcf\") " pod="openshift-console/console-7cbdbcf4b5-rmbsz"
Oct 30 21:50:19.932707 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:19.932687    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Oct 30 21:50:19.936185 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:19.936163    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7cbdbcf4b5-rmbsz"]
Oct 30 21:50:19.937620 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:19.937589    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fdf54bb9-f735-4810-82ee-d803099a1bcf-trusted-ca-bundle\") pod \"console-7cbdbcf4b5-rmbsz\" (UID: \"fdf54bb9-f735-4810-82ee-d803099a1bcf\") " pod="openshift-console/console-7cbdbcf4b5-rmbsz"
Oct 30 21:50:19.987554 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:19.987465    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5jdv\" (UniqueName: \"kubernetes.io/projected/fdf54bb9-f735-4810-82ee-d803099a1bcf-kube-api-access-s5jdv\") pod \"console-7cbdbcf4b5-rmbsz\" (UID: \"fdf54bb9-f735-4810-82ee-d803099a1bcf\") " pod="openshift-console/console-7cbdbcf4b5-rmbsz"
Oct 30 21:50:20.055232 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:20.055177    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7cbdbcf4b5-rmbsz"
Oct 30 21:50:20.433351 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:20.433317    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7cbdbcf4b5-rmbsz"]
Oct 30 21:50:20.446744 ip-10-0-128-245 kubenswrapper[2573]: W1030 21:50:20.446708    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdf54bb9_f735_4810_82ee_d803099a1bcf.slice/crio-b958496c5215e685901b3b9da29749f1fd2fec924a1885b5175591646903879d WatchSource:0}: Error finding container b958496c5215e685901b3b9da29749f1fd2fec924a1885b5175591646903879d: Status 404 returned error can't find the container with id b958496c5215e685901b3b9da29749f1fd2fec924a1885b5175591646903879d
Oct 30 21:50:20.853596 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:20.853565    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ef7ae922-6da7-4817-ba91-35e67b6ac8c5","Type":"ContainerStarted","Data":"80f487ffb5dcf6fc6d14099c357fbc7e27472b4a78950699c63c53d90770256a"}
Oct 30 21:50:20.853696 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:20.853652    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ef7ae922-6da7-4817-ba91-35e67b6ac8c5","Type":"ContainerStarted","Data":"29ceada24258c6d887579990630dc0a842a2ba2a018b98512da626fb7f4b54d9"}
Oct 30 21:50:20.855761 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:20.855732    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-w9tx8" event={"ID":"55619a2e-f6e9-4719-b362-30b69d553967","Type":"ContainerStarted","Data":"ec4a8765260bc1995c56250913155c7c2b2ae3f63e590307076e615611482a6b"}
Oct 30 21:50:20.856513 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:20.856480    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-w9tx8"
Oct 30 21:50:20.858877 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:20.858856    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7cbdbcf4b5-rmbsz" event={"ID":"fdf54bb9-f735-4810-82ee-d803099a1bcf","Type":"ContainerStarted","Data":"e39713154e1dc737f4cfcb4c40f6f2c41e5c329785e99738a71e93998afec246"}
Oct 30 21:50:20.858953 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:20.858895    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7cbdbcf4b5-rmbsz" event={"ID":"fdf54bb9-f735-4810-82ee-d803099a1bcf","Type":"ContainerStarted","Data":"b958496c5215e685901b3b9da29749f1fd2fec924a1885b5175591646903879d"}
Oct 30 21:50:20.970962 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:20.970869    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-w9tx8" podStartSLOduration=66.320793422 podStartE2EDuration="1m10.970851467s" totalImagesPullingTime="4.650058045s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:49:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:50:20.968694105 +0000 UTC m=+72.091318834" watchObservedRunningTime="2026-10-30 21:50:20.970851467 +0000 UTC m=+72.093476197"
Oct 30 21:50:21.109047 ip-10-0-128-245 kubenswrapper[2573]: I1030 21:50:21.108983    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-7cbdbcf4b5-rmbsz" podStartSLOduration=2.108963018 podStartE2EDuration="2.108963018s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:50:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:50:21.101038192 +0000 UTC m=+72.223662921" watchObservedRunningTime="2026-10-30 21:50:21.108963018 +0000 UTC m=+72.231587746"
Oct 30 21:50:22.325249 ip-10-0-128-245 systemd[1]: Stopping Kubernetes Kubelet...
Oct 30 21:50:22.337830 ip-10-0-128-245 systemd[1]: kubelet.service: Deactivated successfully.
Oct 30 21:50:22.338002 ip-10-0-128-245 systemd[1]: Stopped Kubernetes Kubelet.
Oct 30 21:50:22.338216 ip-10-0-128-245 systemd[1]: kubelet.service: Consumed 4.155s CPU time.
Oct 30 21:50:22.362952 ip-10-0-128-245 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 21:50:22.424931 ip-10-0-128-245 kubenswrapper[5678]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 21:50:22.424931 ip-10-0-128-245 kubenswrapper[5678]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 21:50:22.424931 ip-10-0-128-245 kubenswrapper[5678]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 21:50:22.424931 ip-10-0-128-245 kubenswrapper[5678]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 21:50:22.427201 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.427123    5678 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:50:22.427201 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.427201    5678 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:50:22.427201 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.427205    5678 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:50:22.427299 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.427209    5678 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:50:22.427299 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.427212    5678 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:50:22.427299 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.427215    5678 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:50:22.427299 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.427220    5678 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:50:22.427299 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.427224    5678 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:50:22.427299 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.427228    5678 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:50:22.427299 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.427231    5678 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:50:22.427299 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.427233    5678 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:50:22.427299 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.427236    5678 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:50:22.427299 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.427239    5678 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:50:22.427299 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.427241    5678 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:50:22.427299 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.427244    5678 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:50:22.427299 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.427247    5678 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:50:22.427299 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.427249    5678 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:50:22.427299 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.427253    5678 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:50:22.427299 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.427255    5678 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:50:22.427299 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.427258    5678 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:50:22.427299 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.427261    5678 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:50:22.427299 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.427271    5678 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:50:22.427771 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.427274    5678 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:50:22.427771 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.427277    5678 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:50:22.427771 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.427280    5678 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:50:22.427771 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.427283    5678 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:50:22.427771 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.427286    5678 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:50:22.427771 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.427289    5678 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:50:22.427771 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.427291    5678 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:50:22.427771 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.427294    5678 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:50:22.427771 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.427296    5678 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:50:22.427771 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.427299    5678 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:50:22.427771 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.427302    5678 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:50:22.427771 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.427305    5678 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:50:22.427771 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.427308    5678 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:50:22.427771 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.427311    5678 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:50:22.427771 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.427314    5678 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:50:22.427771 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.427316    5678 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:50:22.427771 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.427319    5678 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:50:22.427771 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.427322    5678 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:50:22.427771 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.427324    5678 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:50:22.427771 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.427327    5678 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:50:22.428981 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.427329    5678 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:50:22.428981 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.427332    5678 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:50:22.428981 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.427335    5678 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:50:22.428981 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.427337    5678 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:50:22.428981 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.427340    5678 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:50:22.428981 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.427342    5678 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:50:22.428981 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.427345    5678 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:50:22.428981 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.427348    5678 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:50:22.428981 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.427351    5678 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:50:22.428981 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.427353    5678 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:50:22.428981 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.427356    5678 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:50:22.428981 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.427359    5678 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:50:22.428981 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.427363    5678 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:50:22.428981 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.427366    5678 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:50:22.428981 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.427368    5678 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:50:22.428981 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.427371    5678 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:50:22.428981 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.427374    5678 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:50:22.428981 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.427376    5678 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:50:22.428981 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.427379    5678 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:50:22.428981 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.427381    5678 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:50:22.429480 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.427384    5678 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:50:22.429480 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.427388    5678 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:50:22.429480 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.427392    5678 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:50:22.429480 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.427395    5678 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:50:22.429480 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.427409    5678 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:50:22.429480 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.427412    5678 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:50:22.429480 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.427415    5678 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:50:22.429480 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.427418    5678 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:50:22.429480 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.427420    5678 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:50:22.429480 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.427423    5678 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:50:22.429480 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.427426    5678 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:50:22.429480 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.427429    5678 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:50:22.429480 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.427431    5678 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:50:22.429480 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.427434    5678 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:50:22.429480 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.427437    5678 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:50:22.429480 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.427440    5678 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:50:22.429480 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.427442    5678 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:50:22.429480 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.427445    5678 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:50:22.429480 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.427448    5678 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:50:22.429967 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.427450    5678 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:50:22.429967 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.427453    5678 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:50:22.429967 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.427456    5678 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:50:22.429967 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.427458    5678 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:50:22.429967 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.427461    5678 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:50:22.429967 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.427463    5678 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:50:22.429967 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.427466    5678 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:50:22.429967 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.427469    5678 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:50:22.429967 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.427472    5678 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:50:22.429967 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.427474    5678 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:50:22.429967 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.427477    5678 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:50:22.429967 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.427479    5678 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:50:22.429967 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.427482    5678 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:50:22.429967 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.427484    5678 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:50:22.429967 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.427488    5678 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:50:22.429967 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.427490    5678 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:50:22.429967 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.427493    5678 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:50:22.429967 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.427495    5678 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:50:22.429967 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.427498    5678 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:50:22.430429 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.427501    5678 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:50:22.430429 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.427503    5678 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:50:22.430429 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.427506    5678 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:50:22.430429 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.427509    5678 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:50:22.430429 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.427512    5678 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:50:22.430429 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.427515    5678 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:50:22.430429 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.428272    5678 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:50:22.430429 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.428280    5678 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:50:22.430429 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.428284    5678 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:50:22.430429 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.428287    5678 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:50:22.430429 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.428289    5678 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:50:22.430429 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.428293    5678 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:50:22.430429 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.428296    5678 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:50:22.430429 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.428298    5678 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:50:22.430429 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.428301    5678 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:50:22.430429 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.428303    5678 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:50:22.430429 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.428307    5678 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:50:22.430429 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.428309    5678 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:50:22.430429 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.428312    5678 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:50:22.430885 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.428314    5678 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:50:22.430885 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.428317    5678 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:50:22.430885 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.428320    5678 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:50:22.430885 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.428322    5678 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:50:22.430885 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.428325    5678 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:50:22.430885 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.428327    5678 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:50:22.430885 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.428330    5678 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:50:22.430885 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.428332    5678 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:50:22.430885 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.428335    5678 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:50:22.430885 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.428338    5678 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:50:22.430885 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.428340    5678 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:50:22.430885 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.428343    5678 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:50:22.430885 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.428345    5678 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:50:22.430885 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.428348    5678 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:50:22.430885 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.428350    5678 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:50:22.430885 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.428353    5678 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:50:22.430885 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.428355    5678 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:50:22.430885 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.428358    5678 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:50:22.430885 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.428360    5678 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:50:22.430885 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.428365    5678 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:50:22.431420 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.428368    5678 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:50:22.431420 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.428371    5678 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:50:22.431420 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.428373    5678 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:50:22.431420 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.428376    5678 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:50:22.431420 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.428379    5678 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:50:22.431420 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.428382    5678 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:50:22.431420 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.428385    5678 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:50:22.431420 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.428388    5678 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:50:22.431420 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.428391    5678 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:50:22.431420 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.428393    5678 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:50:22.431420 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.428396    5678 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:50:22.431420 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.428412    5678 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:50:22.431420 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.428416    5678 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:50:22.431420 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.428420    5678 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:50:22.431420 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.428424    5678 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:50:22.431420 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.428428    5678 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:50:22.431420 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.428432    5678 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:50:22.431420 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.428436    5678 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:50:22.431420 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.428440    5678 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:50:22.431420 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.428444    5678 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:50:22.432149 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.428448    5678 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:50:22.432149 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.428452    5678 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:50:22.432149 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.428454    5678 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:50:22.432149 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.428457    5678 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:50:22.432149 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.428460    5678 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:50:22.432149 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.428462    5678 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:50:22.432149 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.428465    5678 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:50:22.432149 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.428467    5678 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:50:22.432149 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.428470    5678 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:50:22.432149 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.428472    5678 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:50:22.432149 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.428475    5678 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:50:22.432149 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.428479    5678 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:50:22.432149 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.428488    5678 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:50:22.432149 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.428492    5678 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:50:22.432149 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.428496    5678 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:50:22.432149 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.428499    5678 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:50:22.432149 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.428503    5678 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:50:22.432149 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.428506    5678 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:50:22.432149 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.428509    5678 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:50:22.432885 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.428512    5678 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:50:22.432885 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.428514    5678 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:50:22.432885 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.428517    5678 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:50:22.432885 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.428519    5678 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:50:22.432885 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.428522    5678 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:50:22.432885 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.428525    5678 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:50:22.432885 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.428527    5678 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:50:22.432885 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.428530    5678 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:50:22.432885 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.428533    5678 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:50:22.432885 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.428535    5678 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:50:22.432885 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.428538    5678 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:50:22.432885 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.428541    5678 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:50:22.432885 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.428543    5678 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:50:22.432885 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.428546    5678 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:50:22.432885 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.428549    5678 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:50:22.432885 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.428552    5678 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:50:22.432885 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.428555    5678 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:50:22.432885 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.428557    5678 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:50:22.432885 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.428560    5678 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:50:22.433588 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.428565    5678 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:50:22.433588 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.428568    5678 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:50:22.433588 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.428571    5678 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:50:22.433588 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.428574    5678 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:50:22.433588 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.428576    5678 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:50:22.433588 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.428581    5678 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:50:22.433588 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.428583    5678 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:50:22.433588 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.428586    5678 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:50:22.433588 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.428589    5678 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:50:22.433588 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.428592    5678 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:50:22.433588 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.428594    5678 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:50:22.433588 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.428597    5678 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:50:22.433588 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.428599    5678 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:50:22.433588 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.428602    5678 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:50:22.433588 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.428605    5678 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:50:22.433588 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.429065    5678 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:50:22.433588 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.429070    5678 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:50:22.433588 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.429073    5678 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:50:22.433588 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.429076    5678 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:50:22.434293 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.429078    5678 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:50:22.434293 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.429081    5678 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:50:22.434293 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.429083    5678 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:50:22.434293 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.429086    5678 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:50:22.434293 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.429089    5678 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:50:22.434293 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.429092    5678 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:50:22.434293 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.429094    5678 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:50:22.434293 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.429099    5678 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:50:22.434293 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.429101    5678 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:50:22.434293 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.429105    5678 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:50:22.434293 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.429109    5678 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:50:22.434293 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.429111    5678 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:50:22.434293 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.429114    5678 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:50:22.434293 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.429116    5678 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:50:22.434293 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.429119    5678 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:50:22.434293 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.429122    5678 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:50:22.434293 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.429125    5678 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:50:22.434293 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.429127    5678 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:50:22.434293 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.429131    5678 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:50:22.434895 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.429133    5678 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:50:22.434895 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.429136    5678 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:50:22.434895 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.429139    5678 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:50:22.434895 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.429141    5678 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:50:22.434895 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.429144    5678 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:50:22.434895 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.429146    5678 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:50:22.434895 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.429149    5678 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:50:22.434895 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.429152    5678 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:50:22.434895 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.429154    5678 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:50:22.434895 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.429157    5678 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:50:22.434895 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.429159    5678 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:50:22.434895 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.429162    5678 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:50:22.434895 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.429165    5678 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:50:22.434895 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.429167    5678 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:50:22.434895 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.429170    5678 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:50:22.434895 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.429172    5678 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:50:22.434895 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.429175    5678 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:50:22.434895 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.429178    5678 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:50:22.434895 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.429181    5678 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:50:22.434895 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.429183    5678 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:50:22.435380 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.429187    5678 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:50:22.435380 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.429191    5678 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:50:22.435380 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.429194    5678 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:50:22.435380 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.429197    5678 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:50:22.435380 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.429202    5678 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:50:22.435380 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.429205    5678 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:50:22.435380 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.429207    5678 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:50:22.435380 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.429210    5678 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:50:22.435380 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.429213    5678 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:50:22.435380 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.429216    5678 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:50:22.435380 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.429219    5678 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:50:22.435380 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.429221    5678 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:50:22.435380 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.429224    5678 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:50:22.435380 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.429226    5678 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:50:22.435380 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.429233    5678 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:50:22.435380 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.429235    5678 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:50:22.435380 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.429238    5678 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:50:22.435380 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.429241    5678 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:50:22.435380 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.429243    5678 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:50:22.435380 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.429245    5678 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:50:22.436018 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.429248    5678 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:50:22.436018 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.429251    5678 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:50:22.436018 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.429253    5678 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:50:22.436018 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.429256    5678 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:50:22.436018 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.429258    5678 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:50:22.436018 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.429261    5678 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:50:22.436018 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.429263    5678 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:50:22.436018 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.429266    5678 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:50:22.436018 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.429268    5678 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:50:22.436018 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.429271    5678 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:50:22.436018 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.429274    5678 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:50:22.436018 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.429277    5678 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:50:22.436018 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.429281    5678 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:50:22.436018 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.429283    5678 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:50:22.436018 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.429286    5678 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:50:22.436018 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.429288    5678 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:50:22.436018 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.429292    5678 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:50:22.436018 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.429294    5678 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:50:22.436018 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.429297    5678 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:50:22.436018 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.429299    5678 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:50:22.436831 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.429302    5678 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:50:22.436831 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.429305    5678 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:50:22.436831 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.429307    5678 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:50:22.436831 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.429310    5678 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:50:22.436831 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.429312    5678 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:50:22.436831 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.429315    5678 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:50:22.436831 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.429317    5678 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:50:22.436831 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.429320    5678 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:50:22.436831 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.429323    5678 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:50:22.436831 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.429325    5678 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:50:22.436831 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.429328    5678 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:50:22.436831 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.429330    5678 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:50:22.436831 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.429333    5678 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:50:22.436831 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.429336    5678 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:50:22.436831 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.429338    5678 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:50:22.436831 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.429341    5678 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:50:22.436831 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.429343    5678 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:50:22.436831 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.429346    5678 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:50:22.436831 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.429349    5678 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:50:22.437293 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.429351    5678 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:50:22.437293 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.429353    5678 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:50:22.437293 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.429356    5678 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:50:22.437293 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.429360    5678 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:50:22.437293 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429468    5678 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 21:50:22.437293 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429478    5678 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 21:50:22.437293 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429484    5678 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 21:50:22.437293 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429489    5678 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 21:50:22.437293 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429494    5678 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 21:50:22.437293 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429497    5678 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 21:50:22.437293 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429502    5678 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 21:50:22.437293 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429506    5678 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 21:50:22.437293 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429509    5678 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 21:50:22.437293 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429512    5678 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 21:50:22.437293 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429517    5678 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 21:50:22.437293 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429521    5678 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 21:50:22.437293 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429524    5678 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 21:50:22.437293 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429527    5678 flags.go:64] FLAG: --cgroup-root=""
Oct 30 21:50:22.437293 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429530    5678 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 21:50:22.437293 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429533    5678 flags.go:64] FLAG: --client-ca-file=""
Oct 30 21:50:22.437293 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429536    5678 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 21:50:22.437293 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429540    5678 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 21:50:22.437293 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429544    5678 flags.go:64] FLAG: --cluster-domain=""
Oct 30 21:50:22.437293 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429547    5678 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 21:50:22.437887 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429550    5678 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 21:50:22.437887 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429553    5678 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 21:50:22.437887 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429556    5678 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 21:50:22.437887 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429560    5678 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 21:50:22.437887 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429563    5678 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 21:50:22.437887 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429566    5678 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 21:50:22.437887 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429569    5678 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 21:50:22.437887 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429573    5678 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 21:50:22.437887 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429576    5678 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 21:50:22.437887 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429579    5678 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 21:50:22.437887 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429582    5678 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 21:50:22.437887 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429585    5678 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 21:50:22.437887 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429589    5678 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 21:50:22.437887 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429593    5678 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 21:50:22.437887 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429596    5678 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 21:50:22.437887 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429599    5678 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 21:50:22.437887 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429603    5678 flags.go:64] FLAG: --enable-server="true"
Oct 30 21:50:22.437887 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429606    5678 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 21:50:22.437887 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429610    5678 flags.go:64] FLAG: --event-burst="100"
Oct 30 21:50:22.437887 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429613    5678 flags.go:64] FLAG: --event-qps="50"
Oct 30 21:50:22.437887 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429616    5678 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 21:50:22.437887 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429619    5678 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 21:50:22.437887 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429624    5678 flags.go:64] FLAG: --eviction-hard=""
Oct 30 21:50:22.437887 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429628    5678 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 21:50:22.438730 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429631    5678 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 21:50:22.438730 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429634    5678 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 21:50:22.438730 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429637    5678 flags.go:64] FLAG: --eviction-soft=""
Oct 30 21:50:22.438730 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429640    5678 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 21:50:22.438730 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429643    5678 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 21:50:22.438730 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429646    5678 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 21:50:22.438730 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429650    5678 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 21:50:22.438730 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429653    5678 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 21:50:22.438730 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429655    5678 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 21:50:22.438730 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429658    5678 flags.go:64] FLAG: --feature-gates=""
Oct 30 21:50:22.438730 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429662    5678 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 21:50:22.438730 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429665    5678 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 21:50:22.438730 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429669    5678 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 21:50:22.438730 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429672    5678 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 21:50:22.438730 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429675    5678 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 21:50:22.438730 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429677    5678 flags.go:64] FLAG: --help="false"
Oct 30 21:50:22.438730 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429681    5678 flags.go:64] FLAG: --hostname-override="ip-10-0-128-245.us-west-2.compute.internal"
Oct 30 21:50:22.438730 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429684    5678 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 21:50:22.438730 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429687    5678 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 21:50:22.438730 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429690    5678 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 21:50:22.438730 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429693    5678 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 21:50:22.438730 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429697    5678 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 21:50:22.438730 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429700    5678 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 21:50:22.439574 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429704    5678 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 21:50:22.439574 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429706    5678 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 21:50:22.439574 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429709    5678 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 21:50:22.439574 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429712    5678 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 21:50:22.439574 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429715    5678 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 21:50:22.439574 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429718    5678 flags.go:64] FLAG: --kube-reserved=""
Oct 30 21:50:22.439574 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429721    5678 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 21:50:22.439574 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429725    5678 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 21:50:22.439574 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429728    5678 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 21:50:22.439574 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429731    5678 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 21:50:22.439574 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429734    5678 flags.go:64] FLAG: --lock-file=""
Oct 30 21:50:22.439574 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429737    5678 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 21:50:22.439574 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429740    5678 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 21:50:22.439574 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429743    5678 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 21:50:22.439574 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429748    5678 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 21:50:22.439574 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429751    5678 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 21:50:22.439574 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429754    5678 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 21:50:22.439574 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429757    5678 flags.go:64] FLAG: --logging-format="text"
Oct 30 21:50:22.439574 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429760    5678 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 21:50:22.439574 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429764    5678 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 21:50:22.439574 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429767    5678 flags.go:64] FLAG: --manifest-url=""
Oct 30 21:50:22.439574 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429770    5678 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 21:50:22.439574 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429774    5678 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 21:50:22.439574 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429777    5678 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 21:50:22.439574 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429784    5678 flags.go:64] FLAG: --max-pods="110"
Oct 30 21:50:22.440343 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429787    5678 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 21:50:22.440343 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429790    5678 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 21:50:22.440343 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429793    5678 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 21:50:22.440343 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429796    5678 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 21:50:22.440343 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429799    5678 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 21:50:22.440343 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429802    5678 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 21:50:22.440343 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429806    5678 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 21:50:22.440343 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429812    5678 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 21:50:22.440343 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429816    5678 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 21:50:22.440343 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429819    5678 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 21:50:22.440343 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429822    5678 flags.go:64] FLAG: --pod-cidr=""
Oct 30 21:50:22.440343 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429825    5678 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 21:50:22.440343 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429828    5678 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 21:50:22.440343 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429831    5678 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 21:50:22.440343 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429835    5678 flags.go:64] FLAG: --port="10250"
Oct 30 21:50:22.440343 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429838    5678 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 21:50:22.440343 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429841    5678 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0f5152ebb45a0e416"
Oct 30 21:50:22.440343 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429845    5678 flags.go:64] FLAG: --qos-reserved=""
Oct 30 21:50:22.440343 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429848    5678 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 21:50:22.440343 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429851    5678 flags.go:64] FLAG: --register-node="true"
Oct 30 21:50:22.440343 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429853    5678 flags.go:64] FLAG: --register-with-taints=""
Oct 30 21:50:22.440343 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429858    5678 flags.go:64] FLAG: --registry-burst="10"
Oct 30 21:50:22.440343 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429861    5678 flags.go:64] FLAG: --registry-qps="5"
Oct 30 21:50:22.440343 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429864    5678 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 21:50:22.440343 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429867    5678 flags.go:64] FLAG: --reserved-memory=""
Oct 30 21:50:22.440343 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429871    5678 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 21:50:22.441172 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429874    5678 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 21:50:22.441172 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429877    5678 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 21:50:22.441172 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429880    5678 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 21:50:22.441172 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429883    5678 flags.go:64] FLAG: --runonce="false"
Oct 30 21:50:22.441172 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429886    5678 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 21:50:22.441172 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429889    5678 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 21:50:22.441172 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429892    5678 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 21:50:22.441172 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429895    5678 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 21:50:22.441172 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429898    5678 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 21:50:22.441172 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429901    5678 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 21:50:22.441172 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429904    5678 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 21:50:22.441172 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429907    5678 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 21:50:22.441172 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429910    5678 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 21:50:22.441172 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429913    5678 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 21:50:22.441172 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429916    5678 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 21:50:22.441172 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429919    5678 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 21:50:22.441172 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429922    5678 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 21:50:22.441172 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429925    5678 flags.go:64] FLAG: --system-cgroups=""
Oct 30 21:50:22.441172 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429928    5678 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 21:50:22.441172 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429933    5678 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 21:50:22.441172 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429938    5678 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 21:50:22.441172 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429941    5678 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 21:50:22.441172 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429945    5678 flags.go:64] FLAG: --tls-min-version=""
Oct 30 21:50:22.441172 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429947    5678 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 21:50:22.441172 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429950    5678 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 21:50:22.441807 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429953    5678 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 21:50:22.441807 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429956    5678 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 21:50:22.441807 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429959    5678 flags.go:64] FLAG: --v="2"
Oct 30 21:50:22.441807 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429963    5678 flags.go:64] FLAG: --version="false"
Oct 30 21:50:22.441807 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429968    5678 flags.go:64] FLAG: --vmodule=""
Oct 30 21:50:22.441807 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429972    5678 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 21:50:22.441807 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.429976    5678 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 21:50:22.441807 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430107    5678 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:50:22.441807 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430111    5678 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:50:22.441807 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430114    5678 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:50:22.441807 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430117    5678 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:50:22.441807 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430120    5678 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:50:22.441807 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430122    5678 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:50:22.441807 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430127    5678 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:50:22.441807 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430129    5678 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:50:22.441807 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430132    5678 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:50:22.441807 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430135    5678 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:50:22.441807 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430137    5678 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:50:22.441807 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430140    5678 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:50:22.441807 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430142    5678 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:50:22.441807 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430145    5678 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:50:22.442294 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430148    5678 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:50:22.442294 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430150    5678 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:50:22.442294 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430153    5678 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:50:22.442294 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430156    5678 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:50:22.442294 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430158    5678 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:50:22.442294 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430161    5678 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:50:22.442294 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430166    5678 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:50:22.442294 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430168    5678 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:50:22.442294 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430171    5678 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:50:22.442294 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430174    5678 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:50:22.442294 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430176    5678 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:50:22.442294 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430179    5678 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:50:22.442294 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430182    5678 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:50:22.442294 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430184    5678 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:50:22.442294 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430187    5678 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:50:22.442294 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430189    5678 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:50:22.442294 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430191    5678 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:50:22.442294 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430194    5678 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:50:22.442294 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430197    5678 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:50:22.442294 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430200    5678 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:50:22.442792 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430202    5678 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:50:22.442792 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430205    5678 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:50:22.442792 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430207    5678 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:50:22.442792 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430210    5678 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:50:22.442792 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430214    5678 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:50:22.442792 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430217    5678 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:50:22.442792 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430219    5678 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:50:22.442792 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430222    5678 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:50:22.442792 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430224    5678 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:50:22.442792 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430227    5678 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:50:22.442792 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430230    5678 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:50:22.442792 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430233    5678 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:50:22.442792 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430237    5678 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:50:22.442792 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430240    5678 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:50:22.442792 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430244    5678 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:50:22.442792 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430246    5678 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:50:22.442792 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430249    5678 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:50:22.442792 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430251    5678 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:50:22.442792 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430257    5678 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:50:22.442792 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430260    5678 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:50:22.443430 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430263    5678 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:50:22.443430 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430266    5678 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:50:22.443430 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430268    5678 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:50:22.443430 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430271    5678 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:50:22.443430 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430273    5678 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:50:22.443430 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430276    5678 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:50:22.443430 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430279    5678 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:50:22.443430 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430281    5678 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:50:22.443430 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430284    5678 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:50:22.443430 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430286    5678 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:50:22.443430 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430289    5678 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:50:22.443430 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430292    5678 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:50:22.443430 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430295    5678 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:50:22.443430 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430297    5678 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:50:22.443430 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430300    5678 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:50:22.443430 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430303    5678 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:50:22.443430 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430306    5678 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:50:22.443430 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430309    5678 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:50:22.443430 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430312    5678 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:50:22.444209 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430314    5678 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:50:22.444209 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430317    5678 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:50:22.444209 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430320    5678 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:50:22.444209 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430323    5678 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:50:22.444209 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430325    5678 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:50:22.444209 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430330    5678 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:50:22.444209 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430334    5678 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:50:22.444209 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430337    5678 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:50:22.444209 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430340    5678 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:50:22.444209 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430342    5678 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:50:22.444209 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430345    5678 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:50:22.444209 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430349    5678 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:50:22.444209 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430351    5678 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:50:22.444209 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430354    5678 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:50:22.444209 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430357    5678 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:50:22.444209 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430359    5678 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:50:22.444209 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430362    5678 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:50:22.444209 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430364    5678 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:50:22.444209 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430367    5678 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:50:22.444209 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430370    5678 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:50:22.445137 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430372    5678 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:50:22.445137 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430375    5678 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:50:22.445137 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430377    5678 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:50:22.445137 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430380    5678 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:50:22.445137 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430383    5678 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:50:22.445137 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430386    5678 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:50:22.445137 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430389    5678 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:50:22.445137 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430391    5678 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:50:22.445137 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430394    5678 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:50:22.445137 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430409    5678 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:50:22.445137 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430413    5678 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:50:22.445137 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430415    5678 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:50:22.445137 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430418    5678 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:50:22.445137 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.430445    5678 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 21:50:22.445137 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430480    5678 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:50:22.445137 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430485    5678 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:50:22.445725 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430488    5678 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:50:22.445725 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430491    5678 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:50:22.445725 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430494    5678 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:50:22.445725 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430498    5678 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:50:22.445725 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430501    5678 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:50:22.445725 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430504    5678 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:50:22.445725 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430507    5678 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:50:22.445725 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430509    5678 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:50:22.445725 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430513    5678 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:50:22.445725 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430516    5678 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:50:22.445725 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430536    5678 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:50:22.445725 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430540    5678 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:50:22.445725 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430543    5678 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:50:22.445725 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430546    5678 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:50:22.445725 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430549    5678 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:50:22.445725 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430552    5678 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:50:22.445725 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430555    5678 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:50:22.445725 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430558    5678 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:50:22.445725 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430560    5678 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:50:22.446303 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430563    5678 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:50:22.446303 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430565    5678 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:50:22.446303 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430568    5678 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:50:22.446303 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430571    5678 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:50:22.446303 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430574    5678 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:50:22.446303 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430577    5678 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:50:22.446303 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430581    5678 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:50:22.446303 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430583    5678 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:50:22.446303 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430586    5678 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:50:22.446303 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430588    5678 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:50:22.446303 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430591    5678 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:50:22.446303 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430594    5678 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:50:22.446303 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430596    5678 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:50:22.446303 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430599    5678 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:50:22.446303 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430602    5678 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:50:22.446303 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430605    5678 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:50:22.446303 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430608    5678 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:50:22.446303 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430612    5678 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:50:22.446303 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430615    5678 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:50:22.446781 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430618    5678 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:50:22.446781 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430621    5678 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:50:22.446781 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430626    5678 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:50:22.446781 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430629    5678 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:50:22.446781 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430632    5678 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:50:22.446781 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430635    5678 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:50:22.446781 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430637    5678 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:50:22.446781 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430640    5678 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:50:22.446781 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430642    5678 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:50:22.446781 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430645    5678 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:50:22.446781 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430647    5678 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:50:22.446781 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430650    5678 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:50:22.446781 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430653    5678 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:50:22.446781 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430655    5678 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:50:22.446781 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430658    5678 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:50:22.446781 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430660    5678 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:50:22.446781 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430663    5678 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:50:22.446781 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430667    5678 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:50:22.446781 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430670    5678 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:50:22.446781 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430674    5678 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:50:22.447248 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430676    5678 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:50:22.447248 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430679    5678 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:50:22.447248 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430681    5678 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:50:22.447248 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430684    5678 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:50:22.447248 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430686    5678 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:50:22.447248 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430689    5678 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:50:22.447248 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430691    5678 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:50:22.447248 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430694    5678 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:50:22.447248 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430697    5678 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:50:22.447248 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430699    5678 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:50:22.447248 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430702    5678 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:50:22.447248 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430704    5678 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:50:22.447248 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430706    5678 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:50:22.447248 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430709    5678 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:50:22.447248 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430713    5678 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:50:22.447248 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430716    5678 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:50:22.447248 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430718    5678 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:50:22.447248 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430721    5678 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:50:22.447248 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430724    5678 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:50:22.447819 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430726    5678 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:50:22.447819 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430729    5678 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:50:22.447819 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430731    5678 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:50:22.447819 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430734    5678 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:50:22.447819 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430737    5678 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:50:22.447819 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430739    5678 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:50:22.447819 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430741    5678 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:50:22.447819 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430744    5678 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:50:22.447819 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430747    5678 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:50:22.447819 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430749    5678 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:50:22.447819 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430752    5678 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:50:22.447819 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430755    5678 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:50:22.447819 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430760    5678 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:50:22.447819 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430762    5678 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:50:22.447819 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430765    5678 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:50:22.447819 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430768    5678 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:50:22.447819 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430770    5678 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:50:22.447819 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430773    5678 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:50:22.447819 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430775    5678 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:50:22.447819 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430778    5678 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:50:22.448384 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430780    5678 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:50:22.448384 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430783    5678 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:50:22.448384 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430785    5678 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:50:22.448384 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430788    5678 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:50:22.448384 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430791    5678 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:50:22.448384 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430793    5678 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:50:22.448384 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.430796    5678 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:50:22.448384 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.435543    5678 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 21:50:22.448384 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.435558    5678 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 21:50:22.448384 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.435575    5678 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 21:50:22.448384 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.435580    5678 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 21:50:22.448384 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.435791    5678 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:50:22.448384 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.435805    5678 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:50:22.448384 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.435810    5678 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:50:22.448384 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.435814    5678 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:50:22.448384 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.435819    5678 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:50:22.448384 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.435823    5678 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:50:22.448384 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.435828    5678 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:50:22.448384 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.435832    5678 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:50:22.448384 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.435836    5678 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:50:22.448880 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.435840    5678 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:50:22.448880 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.435845    5678 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:50:22.448880 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.435849    5678 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:50:22.448880 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.435854    5678 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:50:22.448880 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.435865    5678 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:50:22.448880 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.435869    5678 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:50:22.448880 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.435873    5678 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:50:22.448880 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.435878    5678 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:50:22.448880 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.435882    5678 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:50:22.448880 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.435887    5678 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:50:22.448880 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.435892    5678 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:50:22.448880 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.435897    5678 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:50:22.448880 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.435901    5678 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:50:22.448880 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.435905    5678 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:50:22.448880 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.435910    5678 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:50:22.448880 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.435915    5678 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:50:22.448880 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.435924    5678 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:50:22.448880 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.435936    5678 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:50:22.448880 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.435943    5678 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:50:22.448880 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.435950    5678 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:50:22.449341 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.435955    5678 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:50:22.449341 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.435960    5678 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:50:22.449341 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.435964    5678 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:50:22.449341 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.435968    5678 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:50:22.449341 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.435972    5678 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:50:22.449341 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.435977    5678 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:50:22.449341 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.435981    5678 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:50:22.449341 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.435986    5678 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:50:22.449341 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.435990    5678 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:50:22.449341 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.435999    5678 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:50:22.449341 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.436003    5678 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:50:22.449341 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.436010    5678 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:50:22.449341 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.436016    5678 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:50:22.449341 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.436021    5678 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:50:22.449341 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.436026    5678 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:50:22.449341 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.436030    5678 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:50:22.449341 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.436036    5678 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:50:22.449341 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.436041    5678 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:50:22.449341 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.436045    5678 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:50:22.449814 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.436049    5678 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:50:22.449814 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.436054    5678 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:50:22.449814 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.436064    5678 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:50:22.449814 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.436068    5678 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:50:22.449814 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.436073    5678 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:50:22.449814 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.436077    5678 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:50:22.449814 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.436082    5678 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:50:22.449814 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.436086    5678 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:50:22.449814 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.436091    5678 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:50:22.449814 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.436095    5678 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:50:22.449814 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.436099    5678 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:50:22.449814 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.436104    5678 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:50:22.449814 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.436108    5678 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:50:22.449814 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.436112    5678 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:50:22.449814 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.436116    5678 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:50:22.449814 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.436126    5678 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:50:22.449814 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.436130    5678 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:50:22.449814 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.436134    5678 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:50:22.449814 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.436138    5678 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:50:22.450310 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.436142    5678 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:50:22.450310 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.436147    5678 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:50:22.450310 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.436151    5678 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:50:22.450310 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.436155    5678 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:50:22.450310 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.436159    5678 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:50:22.450310 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.436164    5678 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:50:22.450310 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.436168    5678 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:50:22.450310 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.436172    5678 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:50:22.450310 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.436181    5678 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:50:22.450310 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.436186    5678 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:50:22.450310 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.436190    5678 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:50:22.450310 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.436195    5678 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:50:22.450310 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.436199    5678 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:50:22.450310 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.436203    5678 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:50:22.450310 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.436208    5678 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:50:22.450310 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.436213    5678 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:50:22.450310 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.436217    5678 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:50:22.450310 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.436222    5678 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:50:22.450310 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.436226    5678 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:50:22.450310 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.436231    5678 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:50:22.450893 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.436235    5678 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:50:22.450893 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.436244    5678 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:50:22.450893 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.436249    5678 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:50:22.450893 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.436253    5678 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:50:22.450893 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.436257    5678 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:50:22.450893 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.436262    5678 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:50:22.450893 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.436267    5678 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:50:22.450893 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.436271    5678 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:50:22.450893 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.436275    5678 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:50:22.450893 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.436280    5678 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:50:22.450893 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.436284    5678 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:50:22.450893 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.436288    5678 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:50:22.450893 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.436292    5678 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:50:22.450893 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.436301    5678 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:50:22.450893 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.436305    5678 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:50:22.450893 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.436309    5678 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:50:22.450893 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.436314    5678 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:50:22.450893 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.436318    5678 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:50:22.450893 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.436322    5678 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:50:22.451348 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.436379    5678 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 21:50:22.451348 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437021    5678 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:50:22.451348 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437043    5678 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:50:22.451348 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437049    5678 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:50:22.451348 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437055    5678 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:50:22.451348 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437062    5678 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:50:22.451348 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437067    5678 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:50:22.451348 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437071    5678 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:50:22.451348 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437076    5678 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:50:22.451348 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437080    5678 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:50:22.451348 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437084    5678 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:50:22.451348 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437088    5678 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:50:22.451348 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437093    5678 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:50:22.451348 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437097    5678 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:50:22.451348 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437101    5678 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:50:22.451348 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437106    5678 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:50:22.451837 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437113    5678 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:50:22.451837 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437120    5678 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:50:22.451837 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437125    5678 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:50:22.451837 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437130    5678 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:50:22.451837 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437134    5678 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:50:22.451837 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437138    5678 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:50:22.451837 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437144    5678 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:50:22.451837 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437150    5678 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:50:22.451837 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437154    5678 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:50:22.451837 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437159    5678 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:50:22.451837 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437163    5678 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:50:22.451837 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437169    5678 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:50:22.451837 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437173    5678 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:50:22.451837 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437179    5678 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:50:22.451837 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437184    5678 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:50:22.451837 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437188    5678 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:50:22.451837 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437192    5678 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:50:22.451837 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437197    5678 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:50:22.451837 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437201    5678 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:50:22.452290 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437205    5678 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:50:22.452290 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437210    5678 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:50:22.452290 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437214    5678 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:50:22.452290 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437219    5678 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:50:22.452290 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437223    5678 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:50:22.452290 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437227    5678 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:50:22.452290 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437232    5678 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:50:22.452290 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437236    5678 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:50:22.452290 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437241    5678 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:50:22.452290 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437245    5678 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:50:22.452290 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437249    5678 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:50:22.452290 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437254    5678 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:50:22.452290 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437259    5678 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:50:22.452290 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437265    5678 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:50:22.452290 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437270    5678 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:50:22.452290 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437274    5678 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:50:22.452290 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437278    5678 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:50:22.452290 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437280    5678 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:50:22.452290 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437283    5678 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:50:22.452290 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437286    5678 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:50:22.452778 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437288    5678 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:50:22.452778 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437291    5678 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:50:22.452778 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437294    5678 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:50:22.452778 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437297    5678 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:50:22.452778 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437300    5678 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:50:22.452778 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437303    5678 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:50:22.452778 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437306    5678 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:50:22.452778 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437309    5678 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:50:22.452778 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437311    5678 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:50:22.452778 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437314    5678 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:50:22.452778 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437317    5678 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:50:22.452778 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437319    5678 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:50:22.452778 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437322    5678 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:50:22.452778 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437325    5678 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:50:22.452778 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437328    5678 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:50:22.452778 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437331    5678 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:50:22.452778 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437334    5678 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:50:22.452778 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437336    5678 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:50:22.452778 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437339    5678 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:50:22.453230 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437342    5678 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:50:22.453230 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437345    5678 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:50:22.453230 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437347    5678 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:50:22.453230 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437350    5678 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:50:22.453230 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437353    5678 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:50:22.453230 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437355    5678 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:50:22.453230 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437358    5678 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:50:22.453230 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437362    5678 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:50:22.453230 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437364    5678 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:50:22.453230 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437367    5678 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:50:22.453230 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437369    5678 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:50:22.453230 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437372    5678 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:50:22.453230 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437375    5678 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:50:22.453230 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437377    5678 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:50:22.453230 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437380    5678 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:50:22.453230 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437383    5678 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:50:22.453230 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437385    5678 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:50:22.453230 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437388    5678 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:50:22.453230 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437391    5678 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:50:22.453230 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437394    5678 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:50:22.453785 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437397    5678 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:50:22.453785 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437412    5678 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:50:22.453785 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437414    5678 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:50:22.453785 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437417    5678 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:50:22.453785 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437420    5678 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:50:22.453785 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437422    5678 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:50:22.453785 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437425    5678 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:50:22.453785 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437428    5678 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:50:22.453785 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437430    5678 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:50:22.453785 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437433    5678 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:50:22.453785 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437436    5678 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:50:22.453785 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437438    5678 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:50:22.453785 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437441    5678 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:50:22.453785 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.437482    5678 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 21:50:22.453785 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437522    5678 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:50:22.453785 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437527    5678 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:50:22.454168 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437531    5678 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:50:22.454168 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437534    5678 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:50:22.454168 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437537    5678 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:50:22.454168 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437540    5678 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:50:22.454168 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437543    5678 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:50:22.454168 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437546    5678 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:50:22.454168 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437549    5678 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:50:22.454168 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437551    5678 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:50:22.454168 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437554    5678 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:50:22.454168 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437557    5678 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:50:22.454168 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437559    5678 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:50:22.454168 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437562    5678 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:50:22.454168 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437564    5678 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:50:22.454168 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437567    5678 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:50:22.454168 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437570    5678 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:50:22.454168 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437572    5678 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:50:22.454168 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437575    5678 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:50:22.454168 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437578    5678 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:50:22.454168 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437580    5678 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:50:22.454168 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437583    5678 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:50:22.454662 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437585    5678 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:50:22.454662 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437588    5678 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:50:22.454662 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437591    5678 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:50:22.454662 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437593    5678 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:50:22.454662 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437596    5678 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:50:22.454662 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437598    5678 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:50:22.454662 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437601    5678 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:50:22.454662 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437603    5678 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:50:22.454662 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437606    5678 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:50:22.454662 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437608    5678 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:50:22.454662 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437611    5678 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:50:22.454662 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437613    5678 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:50:22.454662 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437617    5678 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:50:22.454662 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437619    5678 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:50:22.454662 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437622    5678 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:50:22.454662 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437624    5678 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:50:22.454662 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437627    5678 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:50:22.454662 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437629    5678 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:50:22.454662 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437632    5678 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:50:22.455144 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437636    5678 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:50:22.455144 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437640    5678 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:50:22.455144 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437643    5678 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:50:22.455144 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437646    5678 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:50:22.455144 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437648    5678 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:50:22.455144 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437651    5678 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:50:22.455144 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437655    5678 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:50:22.455144 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437658    5678 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:50:22.455144 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437661    5678 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:50:22.455144 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437664    5678 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:50:22.455144 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437666    5678 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:50:22.455144 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437669    5678 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:50:22.455144 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437672    5678 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:50:22.455144 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437675    5678 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:50:22.455144 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437677    5678 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:50:22.455144 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437680    5678 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:50:22.455144 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437682    5678 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:50:22.455144 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437685    5678 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:50:22.455144 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437688    5678 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:50:22.455637 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437690    5678 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:50:22.455637 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437693    5678 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:50:22.455637 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437695    5678 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:50:22.455637 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437699    5678 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:50:22.455637 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437702    5678 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:50:22.455637 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437705    5678 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:50:22.455637 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437707    5678 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:50:22.455637 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437710    5678 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:50:22.455637 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437713    5678 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:50:22.455637 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437716    5678 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:50:22.455637 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437718    5678 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:50:22.455637 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437721    5678 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:50:22.455637 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437724    5678 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:50:22.455637 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437726    5678 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:50:22.455637 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437729    5678 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:50:22.455637 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437731    5678 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:50:22.455637 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437734    5678 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:50:22.455637 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437736    5678 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:50:22.455637 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437739    5678 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:50:22.455637 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437742    5678 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:50:22.456150 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437744    5678 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:50:22.456150 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437747    5678 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:50:22.456150 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437749    5678 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:50:22.456150 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437752    5678 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:50:22.456150 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437754    5678 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:50:22.456150 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437757    5678 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:50:22.456150 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437759    5678 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:50:22.456150 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437762    5678 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:50:22.456150 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437764    5678 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:50:22.456150 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437767    5678 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:50:22.456150 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437769    5678 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:50:22.456150 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437772    5678 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:50:22.456150 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437774    5678 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:50:22.456150 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437777    5678 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:50:22.456150 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437780    5678 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:50:22.456150 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437782    5678 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:50:22.456150 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437785    5678 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:50:22.456150 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437787    5678 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:50:22.456150 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437790    5678 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:50:22.456655 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437792    5678 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:50:22.456655 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437795    5678 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:50:22.456655 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437798    5678 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:50:22.456655 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437802    5678 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:50:22.456655 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437804    5678 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:50:22.456655 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437807    5678 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:50:22.456655 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:22.437809    5678 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:50:22.456655 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.437867    5678 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 21:50:22.456655 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.440848    5678 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 30 21:50:22.456655 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.440910    5678 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 30 21:50:22.456655 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.441090    5678 server.go:1014] "Starting client certificate rotation"
Oct 30 21:50:22.456655 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.441232    5678 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 21:50:22.456655 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.441279    5678 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 21:44:10 +0000 UTC" deadline="2028-05-31 18:53:10.661899163 +0000 UTC"
Oct 30 21:50:22.456655 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.441306    5678 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13893h2m48.220595103s"
Oct 30 21:50:22.456655 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.441561    5678 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 21:50:22.456655 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.442647    5678 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 21:50:22.457041 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.445645    5678 log.go:25] "Validated CRI v1 runtime API"
Oct 30 21:50:22.457041 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.447602    5678 log.go:25] "Validated CRI v1 image API"
Oct 30 21:50:22.457041 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.450364    5678 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 21:50:22.457041 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.452523    5678 fs.go:135] Filesystem UUIDs: map[3ce27a66-195d-48d9-927b-7c384f635460:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 c97bc70d-24ac-4265-9008-ed080b7fcaa1:/dev/nvme0n1p3]
Oct 30 21:50:22.457163 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.452545    5678 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/06de6ffcc0e4da71a6468cd1841f7fb4b2018aedf613cdaed926b55ac3d8b6fe/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/06de6ffcc0e4da71a6468cd1841f7fb4b2018aedf613cdaed926b55ac3d8b6fe/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/11dac6cc6798646aadb4d4e9dc28993e017dd18a9f1a636bf8789dce74026edd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/11dac6cc6798646aadb4d4e9dc28993e017dd18a9f1a636bf8789dce74026edd/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/266940a224ea0a54c1a8a2d9b4e5b33f7b149ab8f92cc7b373d4259ede742beb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/266940a224ea0a54c1a8a2d9b4e5b33f7b149ab8f92cc7b373d4259ede742beb/userdata/shm major:0 minor:392 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2d4d5b332a32797e444f7b86043649ef9e8cd85554203c183f4263d812176bb9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2d4d5b332a32797e444f7b86043649ef9e8cd85554203c183f4263d812176bb9/userdata/shm major:0 minor:401 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3bd252982df6be8fd26cdcd2a455bb58da4c3b32f8bbbdc0df4c8a46ca2826fb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3bd252982df6be8fd26cdcd2a455bb58da4c3b32f8bbbdc0df4c8a46ca2826fb/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4768f1ea299aaff0c7dc0db8d94ec8e94771f4682b4b37ea448a32a4eaa05704/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4768f1ea299aaff0c7dc0db8d94ec8e94771f4682b4b37ea448a32a4eaa05704/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/48c8198c2f2b74b17130ae9678885f3f7d4c981c4bc55458dc71962684f9b0b8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/48c8198c2f2b74b17130ae9678885f3f7d4c981c4bc55458dc71962684f9b0b8/userdata/shm major:0 minor:93 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/54feac0378fd0894bff57a7d55fef92cb31ed8f5b2fcfe974fa7dcb286576b70/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/54feac0378fd0894bff57a7d55fef92cb31ed8f5b2fcfe974fa7dcb286576b70/userdata/shm major:0 minor:398 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5b7fa7c8959b27a9ea4f5af1c51cb119eefb9044902ab5949a2f83aa030ae7cb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5b7fa7c8959b27a9ea4f5af1c51cb119eefb9044902ab5949a2f83aa030ae7cb/userdata/shm major:0 minor:77 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/929ab3025f8a48c7f0741da506a9447ead41f7e897a54256d70954ade722c613/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/929ab3025f8a48c7f0741da506a9447ead41f7e897a54256d70954ade722c613/userdata/shm major:0 minor:212 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/99e016196628ee432af4e398621a5760c9b2739cc2b5a123742aa8523e334b05/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/99e016196628ee432af4e398621a5760c9b2739cc2b5a123742aa8523e334b05/userdata/shm major:0 minor:287 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a7fc187ce460fbaeec60b28f3c313837ba07a8341b8639946b2382e42dde1919/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a7fc187ce460fbaeec60b28f3c313837ba07a8341b8639946b2382e42dde1919/userdata/shm major:0 minor:339 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b80c4122abd5858b56886f4d518ab4c81848d68f3a7fd253c2ebc4930ac56b10/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b80c4122abd5858b56886f4d518ab4c81848d68f3a7fd253c2ebc4930ac56b10/userdata/shm major:0 minor:289 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b8fd19e59bf3f63ce8b861120a1c5a46871e91627d90f3725696a99fc1e6f4f5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b8fd19e59bf3f63ce8b861120a1c5a46871e91627d90f3725696a99fc1e6f4f5/userdata/shm major:0 minor:210 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b958496c5215e685901b3b9da29749f1fd2fec924a1885b5175591646903879d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b958496c5215e685901b3b9da29749f1fd2fec924a1885b5175591646903879d/userdata/shm major:0 minor:425 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ca41b3353f4cff5c25e5091dd9731c19f4a1577f6aceaf86e9b91db22ba7f5e1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ca41b3353f4cff5c25e5091dd9731c19f4a1577f6aceaf86e9b91db22ba7f5e1/userdata/shm major:0 minor:214 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d96925669f74a34a6a8e0245a59131261ec4d0a7c88335b9eced7a1f596f1b04/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d96925669f74a34a6a8e0245a59131261ec4d0a7c88335b9eced7a1f596f1b04/userdata/shm major:0 minor:403 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/da9e0ba8a0e86180f7612fe2a827ab57b9c0afcdf1a087ac9140a2b5de077636/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/da9e0ba8a0e86180f7612fe2a827ab57b9c0afcdf1a087ac9140a2b5de077636/userdata/shm major:0 minor:208 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/dc24a46b097bebb5444228144f254835785f13e6c1da310ba365dff351a254f6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/dc24a46b097bebb5444228144f254835785f13e6c1da310ba365dff351a254f6/userdata/shm major:0 minor:317 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ef9db6f7956a084b994e446966ee810675c93075b4589ea22626894079d461b1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ef9db6f7956a084b994e446966ee810675c93075b4589ea22626894079d461b1/userdata/shm major:0 minor:70 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f6a186b19f8e7b056b1e01d0d6141ef6ea402c1dbb9a2d946a89669c6b60f63f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f6a186b19f8e7b056b1e01d0d6141ef6ea402c1dbb9a2d946a89669c6b60f63f/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f81df74784eeabd8abcee1495fda7eb479bb06f0826ce1b438d3632d3bd104b4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f81df74784eeabd8abcee1495fda7eb479bb06f0826ce1b438d3632d3bd104b4/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ff7b45b007ad888b3545c4a198046fb18b188e5a1e0e7854aa6c5a60dc784808/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ff7b45b007ad888b3545c4a198046fb18b188e5a1e0e7854aa6c5a60dc784808/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ffcebcbe6a69d7a5b1acce836ecf6509f9155e88b6bd05c7b2f2dc9c025ae664/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ffcebcbe6a69d7a5b1acce836ecf6509f9155e88b6bd05c7b2f2dc9c025ae664/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0ad59239-edcc-46a4-a5a1-1debb63e3a5d/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/0ad59239-edcc-46a4-a5a1-1debb63e3a5d/volumes/kubernetes.io~secret/agent-certs major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/32d63b18-0fcb-4e4a-a24c-ee044dcb5a90/volumes/kubernetes.io~projected/kube-api-access-pxd2f:{mountpoint:/var/lib/kubelet/pods/32d63b18-0fcb-4e4a-a24c-ee044dcb5a90/volumes/kubernetes.io~projected/kube-api-access-pxd2f major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/41fe6d63-9316-4916-b042-87ba7e602167/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/41fe6d63-9316-4916-b042-87ba7e602167/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/41fe6d63-9316-4916-b042-87ba7e602167/volumes/kubernetes.io~projected/kube-api-access-2hghq:{mountpoint:/var/lib/kubelet/pods/41fe6d63-9316-4916-b042-87ba7e602167/volumes/kubernetes.io~projected/kube-api-access-2hghq major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/41fe6d63-9316-4916-b042-87ba7e602167/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/41fe6d63-9316-4916-b042-87ba7e602167/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/49ff8cb6-54de-4e8c-9a35-634f9a1ef29b/volumes/kubernetes.io~projected/kube-api-access-z7qxb:{mountpoint:/var/lib/kubelet/pods/49ff8cb6-54de-4e8c-9a35-634f9a1ef29b/volumes/kubernetes.io~projected/kube-api-access-z7qxb major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/49ff8cb6-54de-4e8c-9a35-634f9a1ef29b/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/49ff8cb6-54de-4e8c-9a35-634f9a1ef29b/volumes/kubernetes.io~secret/metrics-tls major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5138fe8c-9927-4d05-8fe5-5556072ed595/volume-subpaths/cluster-tls-config/alertmanager/10:{mountpoint:/var/lib/kubelet/pods/5138fe8c-9927-4d05-8fe5-5556072ed595/volume-subpaths/cluster-tls-config/alertmanager/10 major:0 minor:295 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5138fe8c-9927-4d05-8fe5-5556072ed595/volume-subpaths/web-config/alertmanager/9:{mountpoint:/var/lib/kubelet/pods/5138fe8c-9927-4d05-8fe5-5556072ed595/volume-subpaths/web-config/alertmanager/9 major:0 minor:299 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5138fe8c-9927-4d05-8fe5-5556072ed595/volume-subpaths/web-config/config-reloader/6:{mountpoint:/var/lib/kubelet/pods/5138fe8c-9927-4d05-8fe5-5556072ed595/volume-subpaths/web-config/config-reloader/6 major:0 minor:299 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5138fe8c-9927-4d05-8fe5-5556072ed595/volume-subpaths/web-config/init-config-reloader/6:{mountpoint:/var/lib/kubelet/pods/5138fe8c-9927-4d05-8fe5-5556072ed595/volume-subpaths/web-config/init-config-reloader/6 major:0 minor:299 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5138fe8c-9927-4d05-8fe5-5556072ed595/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/5138fe8c-9927-4d05-8fe5-5556072ed595/volumes/kubernetes.io~empty-dir/config-out major:0 minor:302 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5138fe8c-9927-4d05-8fe5-5556072ed595/volumes/kubernetes.io~projected/kube-api-access-6v9rf:{mountpoint:/var/lib/kubelet/pods/5138fe8c-9927-4d05-8fe5-5556072ed595/volumes/kubernetes.io~projected/kube-api-access-6v9rf major:0 minor:304 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5138fe8c-9927-4d05-8fe5-5556072ed595/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/5138fe8c-9927-4d05-8fe5-5556072ed595/volumes/kubernetes.io~projected/tls-assets major:0 minor:303 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5138fe8c-9927-4d05-8fe5-5556072ed595/volumes/kubernetes.io~secret/cluster-tls-config:{mountpoint:/var/lib/kubelet/pods/5138fe8c-9927-4d05-8fe5-5556072ed595/volumes/kubernetes.io~secret/cluster-tls-config major:0 minor:295 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5138fe8c-9927-4d05-8fe5-5556072ed595/volumes/kubernetes.io~secret/config-volume:{mountpoint:/var/lib/kubelet/pods/5138fe8c-9927-4d05-8fe5-5556072ed595/volumes/kubernetes.io~secret/config-volume major:0 minor:298 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5138fe8c-9927-4d05-8fe5-5556072ed595/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/5138fe8c-9927-4d05-8fe5-5556072ed595/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy major:0 minor:301 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5138fe8c-9927-4d05-8fe5-5556072ed595/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric:{mountpoint:/var/lib/kubelet/pods/5138fe8c-9927-4d05-8fe5-5556072ed595/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric major:0 minor:297 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5138fe8c-9927-4d05-8fe5-5556072ed595/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/5138fe8c-9927-4d05-8fe5-5556072ed595/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web major:0 minor:300 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5138fe8c-9927-4d05-8fe5-5556072ed595/volumes/kubernetes.io~secret/secret-alertmanager-main-tls:{mountpoint:/var/lib/kubelet/pods/5138fe8c-9927-4d05-8fe5-5556072ed595/volumes/kubernetes.io~secret/secret-alertmanager-main-tls major:0 minor:296 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5138fe8c-9927-4d05-8fe5-5556072ed595/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/5138fe8c-9927-4d05-8fe5-5556072ed595/volumes/kubernetes.io~secret/web-config major:0 minor:299 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/516573f9-4ada-4868-b885-f61b9640e6bf/volumes/kubernetes.io~projected/kube-api-access-66wt8:{mountpoint:/var/lib/kubelet/pods/516573f9-4ada-4868-b885-f61b9640e6bf/volumes/kubernetes.io~projected/kube-api-access-66wt8 major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/516573f9-4ada-4868-b885-f61b9640e6bf/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/516573f9-4ada-4868-b885-f61b9640e6bf/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/55619a2e-f6e9-4719-b362-30b69d553967/volumes/kubernetes.io~projected/kube-api-access-j6x28:{mountpoint:/var/lib/kubelet/pods/55619a2e-f6e9-4719-b362-30b69d553967/volumes/kubernetes.io~projected/kube-api-access-j6x28 major:0 minor:400 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5f468f3c-bf3d-437c-8ec8-a5d56f448d99/volumes/kubernetes.io~projected/kube-api-access-rgn7x:{mountpoint:/var/lib/kubelet/pods/5f468f3c-bf3d-437c-8ec8-a5d56f448d99/volumes/kubernetes.io~projected/kube-api-access-rgn7x major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/67b50e2a-4d01-4d5f-a5d1-7e80ee62642a/volumes/kubernetes.io~projected/kube-api-access-fn582:{mountpoint:/var/lib/kubelet/pods/67b50e2a-4d01-4d5f-a5d1-7e80ee62642a/volumes/kubernetes.io~projected/kube-api-access-fn582 major:0 minor:244 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/67b50e2a-4d01-4d5f-a5d1-7e80ee62642a/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/67b50e2a-4d01-4d5f-a5d1-7e80ee62642a/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config major:0 minor:238 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/67b50e2a-4d01-4d5f-a5d1-7e80ee62642a/volumes/kubernetes.io~secret/prometheus-operator-tls:{mountpoint:/var/lib/kubelet/pods/67b50e2a-4d01-4d5f-a5d1-7e80ee62642a/volumes/kubernetes.io~secret/prometheus-operator-tls major:0 minor:237 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/69e218ec-ab5e-4e09-8444-e53a710c53bf/volumes/kubernetes.io~projected/kube-api-access-gv6wf:{mountpoint:/var/lib/kubelet/pods/69e218ec-ab5e-4e09-8444-e53a710c53bf/volumes/kubernetes.io~projected/kube-api-access-gv6wf major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/69e218ec-ab5e-4e09-8444-e53a710c53bf/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/69e218ec-ab5e-4e09-8444-e53a710c53bf/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/69e218ec-ab5e-4e09-8444-e53a710c53bf/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/69e218ec-ab5e-4e09-8444-e53a710c53bf/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7a6305de-8fd1-4d3e-9628-8c4cda07f379/volumes/kubernetes.io~projected/kube-api-access-lwvcm:{mountpoint:/var/lib/kubelet/pods/7a6305de-8fd1-4d3e-9628-8c4cda07f379/volumes/kubernetes.io~projected/kube-api-access-lwvcm major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/82f75dd5-40cd-4cb3-9111-6213d6b43679/volumes/kubernetes.io~projected/kube-api-access-b97qt:{mountpoint:/var/lib/kubelet/pods/82f75dd5-40cd-4cb3-9111-6213d6b43679/volumes/kubernetes.io~projected/kube-api-access-b97qt major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/87c44883-54d0-4a05-b94b-d3c2835a03e1/volumes/kubernetes.io~projected/kube-api-access-cdznq:{mountpoint:/var/lib/kubelet/pods/87c44883-54d0-4a05-b94b-d3c2835a03e1/volumes/kubernetes.io~projected/kube-api-access-cdznq major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/87c44883-54d0-4a05-b94b-d3c2835a03e1/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/87c44883-54d0-4a05-b94b-d3c2835a03e1/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/924394f7-5b3b-4767-a1ff-f1d310ece4ed/volumes/kubernetes.io~projected/kube-api-access-qnt2l:{mountpoint:/var/lib/kubelet/pods/924394f7-5b3b-4767-a1ff-f1d310ece4ed/volumes/kubernetes.io~projected/kube-api-access-qnt2l major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a3459ebb-2fc2-4a82-b9c0-de5542d667e1/volumes/kubernetes.io~projected/kube-api-access-6hdbq:{mountpoint:/var/lib/kubelet/pods/a3459ebb-2fc2-4a82-b9c0-de5542d667e1/volumes/kubernetes.io~projected/kube-api-access-6hdbq major:0 minor:338 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a3459ebb-2fc2-4a82-b9c0-de5542d667e1/volumes/kubernetes.io~secret/client-ca-bundle:{mountpoint:/var/lib/kubelet/pods/a3459ebb-2fc2-4a82-b9c0-de5542d667e1/volumes/kubernetes.io~secret/client-ca-bundle major:0 minor:337 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a3459ebb-2fc2-4a82-b9c0-de5542d667e1/volumes/kubernetes.io~secret/secret-metrics-server-client-certs:{mountpoint:/var/lib/kubelet/pods/a3459ebb-2fc2-4a82-b9c0-de5542d667e1/volumes/kubernetes.io~secret/secret-metrics-server-client-certs major:0 minor:336 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a3459ebb-2fc2-4a82-b9c0-de5542d667e1/volumes/kubernetes.io~secret/secret-metrics-server-tls:{mountpoint:/var/lib/kubelet/pods/a3459ebb-2fc2-4a82-b9c0-de5542d667e1/volumes/kubernetes.io~secret/secret-metrics-server-tls major:0 minor:335 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a6980f40-89cf-4f3d-8cd5-91969ae5a276/volumes/kubernetes.io~projected/kube-api-access-czfxj:{mountpoint:/var/lib/kubelet/pods/a6980f40-89cf-4f3d-8cd5-91969ae5a276/volumes/kubernetes.io~projected/kube-api-access-czfxj major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a6980f40-89cf-4f3d-8cd5-91969ae5a276/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/a6980f40-89cf-4f3d-8cd5-91969ae5a276/volumes/kubernetes.io~secret/metrics-certs major:0 minor:395 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b954bab2-9dd7-4b39-bda6-3fec87381aff/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/b954bab2-9dd7-4b39-bda6-3fec87381aff/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:394 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d947d3db-ab6e-4874-8817-f8c74be138a1/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/d947d3db-ab6e-4874-8817-f8c74be138a1/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d947d3db-ab6e-4874-8817-f8c74be138a1/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/d947d3db-ab6e-4874-8817-f8c74be138a1/volumes/kubernetes.io~empty-dir/tmp major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d947d3db-ab6e-4874-8817-f8c74be138a1/volumes/kubernetes.io~projected/kube-api-access-mkkmr:{mountpoint:/var/lib/kubelet/pods/d947d3db-ab6e-4874-8817-f8c74be138a1/volumes/kubernetes.io~projected/kube-api-access-mkkmr major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dec1cd54-fec6-4c12-9d37-a1b0b1a7ffc0/volumes/kubernetes.io~projected/kube-api-access-wvbdq:{mountpoint:/var/lib/kubelet/pods/dec1cd54-fec6-4c12-9d37-a1b0b1a7ffc0/volumes/kubernetes.io~projected/kube-api-access-wvbdq major:0 minor:283 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dec1cd54-fec6-4c12-9d37-a1b0b1a7ffc0/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/dec1cd54-fec6-4c12-9d37-a1b0b1a7ffc0/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:282 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dec1cd54-fec6-4c12-9d37-a1b0b1a7ffc0/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/dec1cd54-fec6-4c12-9d37-a1b0b1a7ffc0/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:278 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ef7ae922-6da7-4817-ba91-35e67b6ac8c5/volume-subpaths/web-config/config-reloader/2:{mountpoint:/var/lib/kubelet/pods/ef7ae922-6da7-4817-ba91-35e67b6ac8c5/volume-subpaths/web-config/config-reloader/2 major:0 minor:377 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ef7ae922-6da7-4817-ba91-35e67b6ac8c5/volume-subpaths/web-config/init-config-reloader/2:{mountpoint:/var/lib/kubelet/pods/ef7ae922-6da7-4817-ba91-35e67b6ac8c5/volume-subpaths/web-config/init-config-reloader/2 major:0 minor:377 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ef7ae922-6da7-4817-ba91-35e67b6ac8c5/volume-subpaths/web-config/prometheus/15:{mountpoint:/var/lib/kubelet/pods/ef7ae922-6da7-4817-ba91-35e67b6ac8c5/volume-subpaths/web-config/prometheus/15 major:0 minor:377 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ef7ae922-6da7-4817-ba91-35e67b6ac8c5/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/ef7ae922-6da7-4817-ba91-35e67b6ac8c5/volumes/kubernetes.io~empty-dir/config-out major:0 minor:390 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ef7ae922-6da7-4817-ba91-35e67b6ac8c5/volumes/kubernetes.io~projected/kube-api-access-cr8x2:{mountpoint:/var/lib/kubelet/pods/ef7ae922-6da7-4817-ba91-35e67b6ac8c5/volumes/kubernetes.io~projected/kube-api-access-cr8x2 major:0 minor:391 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ef7ae922-6da7-4817-ba91-35e67b6ac8c5/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/ef7ae922-6da7-4817-ba91-35e67b6ac8c5/volumes/kubernetes.io~projected/tls-assets major:0 minor:388 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ef7ae922-6da7-4817-ba91-35e67b6ac8c5/volumes/kubernetes.io~secret/config:{mountpoint:/var/lib/kubelet/pods/ef7ae922-6da7-4817-ba91-35e67b6ac8c5/volumes/kubernetes.io~secret/config major:0 minor:389 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ef7ae922-6da7-4817-ba91-35e67b6ac8c5/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/ef7ae922-6da7-4817-ba91-35e67b6ac8c5/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:384 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ef7ae922-6da7-4817-ba91-35e67b6ac8c5/volumes/kubernetes.io~secret/secret-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/ef7ae922-6da7-4817-ba91-35e67b6ac8c5/volumes/kubernetes.io~secret/secret-kube-rbac-proxy major:0 minor:386 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ef7ae922-6da7-4817-ba91-35e67b6ac8c5/volumes/kubernetes.io~secret/secret-metrics-client-certs:{mountpoint:/var/lib/kubelet/pods/ef7ae922-6da7-4817-ba91-35e67b6ac8c5/volumes/kubernetes.io~secret/secret-metrics-client-certs major:0 minor:383 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ef7ae922-6da7-4817-ba91-35e67b6ac8c5/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/ef7ae922-6da7-4817-ba91-35e67b6ac8c5/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web major:0 minor:385 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ef7ae922-6da7-4817-ba91-35e67b6ac8c5/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls:{mountpoint:/var/lib/kubelet/pods/ef7ae922-6da7-4817-ba91-35e67b6ac8c5/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls major:0 minor:387 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ef7ae922-6da7-4817-ba91-35e67b6ac8c5/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls:{mountpoint:/var/lib/kubelet/pods/ef7ae922-6da7-4817-ba91-35e67b6ac8c5/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls major:0 minor:381 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ef7ae922-6da7-4817-ba91-35e67b6ac8c5/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file:{mountpoint:/var/lib/kubelet/pods/ef7ae922-6da7-4817-ba91-35e67b6ac8c5/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file major:0 minor:382 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ef7ae922-6da7-4817-ba91-35e67b6ac8c5/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/ef7ae922-6da7-4817-ba91-35e67b6ac8c5/volumes/kubernetes.io~secret/web-config major:0 minor:377 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f22e4514-ac6d-481d-8afe-6e2e4cdc3aa1/volumes/kubernetes.io~projected/kube-api-access-8w8vm:{mountpoint:/var/lib/kubelet/pods/f22e4514-ac6d-481d-8afe-6e2e4cdc3aa1/volumes/kubernetes.io~projected/kube-api-access-8w8vm major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f938a93b-29dd-4e7d-b8e9-444df68acf03/volumes/kubernetes.io~projected/kube-api-access-6jt5t:{mountpoint:/var/lib/kubelet/pods/f938a93b-29dd-4e7d-b8e9-444df68acf03/volumes/kubernetes.io~projected/kube-api-access-6jt5t major:0 minor:286 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f938a93b-29dd-4e7d-b8e9-444df68acf03/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/f938a93b-29dd-4e7d-b8e9-444df68acf03/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:285 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f938a93b-29dd-4e7d-b8e9-444df68acf03/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/f938a93b-29dd-4e7d-b8e9-444df68acf03/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:284 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fdf54bb9-f735-4810-82ee-d803099a1bcf/volumes/kubernetes.io~projected/kube-api-access-s5jdv:{mountpoint:/var/lib/kubelet/pods/fdf54bb9-f735-4810-82ee-d803099a1bcf/volumes/kubernetes.io~projected/kube-api-access-s5jdv major:0 minor:424 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fdf54bb9-f735-4810-82ee-d803099a1bcf/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/fdf54bb9-f735-4810-82ee-d803099a1bcf/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:423 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fdf54bb9-f735-4810-82ee-d803099a1bcf/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/fdf54bb9-f735-4810-82ee-d803099a1bcf/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:419 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/4013d87388a63df5fd005025099feb34dc27529964bd59b55d886563bf14ded9/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/7a1dccb3f9bd91a75306e818d1f03f4ea224061b66281bbce26500741d6ccbb5/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/6f53e3bff4c71c8c9944eed431e69f13e6092beaf6d65810e59fc62bbd688ef8/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/7bf96c738fb3179041910a061b65f081a6015d035357d9c422a6a2d3b33a0e92/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-111:{mountpoint:/var/lib/containers/storage/overlay/c6fcaf858df46d2b619b61393388cc5550871d17fb62e5fdc5f8dfb66291059f/merged major:0 minor:111 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/7140ca4950a9067b664aeb8d4e5b67601c48547d4cb33062596d65df839e7e70/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/f4328dff2543ee5544afaa05642b2d1599a91d7146580808b658fe21b8ef7b50/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/f56b6d9daaa4ca521bb9f70dea70f8c5e4ea59409adf520633d9bfe783941f39/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/916cdf3f438a108338ca0c73194f203e983ec28267c0cf941beb498f97e91f79/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/8ce0479404192755754d718cd8575bc2b261caa73beaf96259578b8a689ac17e/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-142:{mountpoint:/var/lib/containers/storage/overlay/c3746875e89f9dfd03acff70e255d1f2db0dceceb52beed5e42c866d37ed3690/merged major:0 minor:142 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/f3d62881752d25941b5aab38d363d40b0e3a32b5e3bfe1af8b290f41169bc654/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/5504792263beae8ac9ed46717a0bf2d0feefd47ca339d3e84d206f2a2e7938f8/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/f516f8a6c26f00359146a3db7f79effeda40afca49946b71b623328074951926/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/0be0e7445f1d01a453814600a1317ce7ed66c8dacebe0d0964f9d2dffd5b4626/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/c403eae76046f8958a5ccf0ab04664582ab971c8dbd24195e545a3dd91791d2c/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/05d8af417f3762d258360c8cc0142a443fad9e94831d2240d6f6a7e900945af8/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/c118881c96ab65cc1cf4d0f94f158b1959e61fe6550f4320667d5e2adc6daee6/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/022dc46e3275981b46ae8d3839c8ea5ef4f51ebe4c410369e6241a1d0426b556/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/226497bb765b95e2de654ccab25aa8d28017ba1bafec1bd9239aaef3bed8cae9/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/eb111a5b5f068a343b68e0d28965894d4b108755779b602f1ce8554d6e9baaf2/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-216:{mountpoint:/var/lib/containers/storage/overlay/b5fe94e408452e4287ae9d63c958c3bffc10b8289f02f81d03f66fff21d7eba6/merged major:0 minor:216 fsType:overlay blockSize:0} overlay_0-218:{mountpoint:/var/lib/containers/storage/overlay/01795414b46eeedce566335ce453e10f6e7ce632d6b1d516635c2ba244f7b70e/merged major:0 minor:218 fsType:overlay blockSize:0} overlay_0-220:{mountpoint:/var/lib/containers/storage/overlay/42b46f8ff3e7cbb613c40acc4720c185f64f98f48bdecefd092da7b8097b7f45/merged major:0 minor:220 fsType:overlay blockSize:0} overlay_0-222:{mountpoint:/var/lib/containers/storage/overlay/a0b844bc17378b323c6143bc629510e2309bb9121545e164c1c17e0405f45752/merged major:0 minor:222 fsType:overlay blockSize:0} overlay_0-227:{mountpoint:/var/lib/containers/storage/overlay/695a12a5360c8e08392ea206e06d37fffa66eb86b0a5f6fa3923ca82f204d62d/merged major:0 minor:227 fsType:overlay blockSize:0} overlay_0-229:{mountpoint:/var/lib/containers/storage/overlay/d44e25b10bc5e717544ad02ac8e4c2672c523f81d03538842e9fb795614b1ba2/merged major:0 minor:229 fsType:overlay blockSize:0} overlay_0-232:{mountpoint:/var/lib/containers/storage/overlay/dc869e1fac2cc6bca98da684dec633f37f022739b1a15562cce57a57c65975fa/merged major:0 minor:232 fsType:overlay blockSize:0} overlay_0-239:{mountpoint:/var/lib/containers/storage/overlay/ac78d4d73737b8fc376c52f5659ae662e2004744cad8299ff6e57adb55af945e/merged major:0 minor:239 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/dae3d7d255f32ab769124dc92525b9ee08e5c1990494260db770062d6fb73763/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-253:{mountpoint:/var/lib/containers/storage/overlay/52adf79b8fdf4481bedf29fa0f4cfc44b4e2c777424a6487f950f6f7e7e61448/merged major:0 minor:253 fsType:overlay blockSize:0} overlay_0-255:{mountpoint:/var/lib/containers/storage/overlay/007d79d6b4acea8bce1b4e222280211692ef76b6dd8b41193b4db600afa84f2f/merged major:0 minor:255 fsType:overlay blockSize:0} overlay_0-261:{mountpoint:/var/lib/containers/storage/overlay/f81c6ee48b45857ef8e04ede0669e3b5b54ec009d64db1a62e36911357ff2336/merged major:0 minor:261 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/58b411e0f66e7709c26994cd00f62dd90a7df9969e3a4f753a659f2cde9699dd/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-272:{mountpoint:/var/lib/containers/storage/overlay/7ac13d04bc5900d8613470387e0bf9cf012301f5643144d11f0d97f5a93ffae9/merged major:0 minor:272 fsType:overlay blockSize:0} overlay_0-291:{mountpoint:/var/lib/containers/storage/overlay/da25dadfb581b3912c541a96c2607a283f6a41a521eb41fed2280275c4e710ea/merged major:0 minor:291 fsType:overlay blockSize:0} overlay_0-293:{mountpoint:/var/lib/containers/storage/overlay/a42c49071d98f50a421ef3cf8ff4ca011f24835f6da231805950b6d59216d042/merged major:0 minor:293 fsType:overlay blockSize:0} overlay_0-305:{mountpoint:/var/lib/containers/storage/overlay/7a2dd5517e734e7e750dbf6f8c3edc7717637bcba43d00fb8f0adee3177c17ec/merged major:0 minor:305 fsType:overlay blockSize:0} overlay_0-307:{mountpoint:/var/lib/containers/storage/overlay/ed2970835c399877659a0cd094eb8705b31ee5e38822f37a71a127742fccdaa4/merged major:0 minor:307 fsType:overlay blockSize:0} overlay_0-313:{mountpoint:/var/lib/containers/storage/overlay/effb4821d309070b4aba7c60f22b37a2566131250c67936f67f8175de3882fde/merged major:0 minor:313 fsType:overlay blockSize:0} overlay_0-320:{mountpoint:/var/lib/containers/storage/overlay/7eb3dcf9050086e931368ed047f1ceeed28b89060e3a3b43e9018a04b8424e65/merged major:0 minor:320 fsType:overlay blockSize:0} overlay_0-324:{mountpoint:/var/lib/containers/storage/overlay/cc58d8b0dac0e9f2c7c10f2eaa8621eaaf6500fd925512d13a9156ec44bbccb7/merged major:0 minor:324 fsType:overlay blockSize:0} overlay_0-330:{mountpoint:/var/lib/containers/storage/overlay/ec50008d29347bacab29012c83871e50731a8472be15f414196133a941e0c065/merged major:0 minor:330 fsType:overlay blockSize:0} overlay_0-341:{mountpoint:/var/lib/containers/storage/overlay/0563487df9be4f54520f4fc898820d036ba96bebd36a5fc28ddedde3d1a1abd9/merged major:0 minor:341 fsType:overlay blockSize:0} overlay_0-343:{mountpoint:/var/lib/containers/storage/overlay/352a0a68306d3224b75365ccd138e05c4faa123fb0a19dba33dcf01c3499d604/merged major:0 minor:343 fsType:overlay blockSize:0} overlay_0-345:{mountpoint:/var/lib/containers/storage/overlay/fd00e2b9739cbb0b601453627bc3d968f4c02690e559ad366aa8731505b6c962/merged major:0 minor:345 fsType:overlay blockSize:0} overlay_0-351:{mountpoint:/var/lib/containers/storage/overlay/76302aa6abb674a9905f011d86437dbb88b5a616a7e12ba1e2bfa971c8a4e896/merged major:0 minor:351 fsType:overlay blockSize:0} overlay_0-356:{mountpoint:/var/lib/containers/storage/overlay/779f03318abbf7e73033331e059635b3e93eb9e0a934d47a7887e9bbc094263b/merged major:0 minor:356 fsType:overlay blockSize:0} overlay_0-358:{mountpoint:/var/lib/containers/storage/overlay/8a66de9a35c282e3c1dbfd42a39b712b12f4b43ffabc766bfb9c8ac5f1424f83/merged major:0 minor:358 fsType:overlay blockSize:0} overlay_0-359:{mountpoint:/var/lib/containers/storage/overlay/806e461f2cb98dc1449a18c000a133e209cf25f73d68572f68c8d0f596beae9a/merged major:0 minor:359 fsType:overlay blockSize:0} overlay_0-372:{mountpoint:/var/lib/containers/storage/overlay/4f85ac8bf8d25991967afb80d8d96851d56d447d8a4c6133710bc14acec9caae/merged major:0 minor:372 fsType:overlay blockSize:0} overlay_0-396:{mountpoint:/var/lib/containers/storage/overlay/a3bb37af6afd473e1d68e16db13370eccdbd5c37c022b7ab797e50fcf5acc439/merged major:0 minor:396 fsType:overlay blockSize:0} overlay_0-405:{mountpoint:/var/lib/containers/storage/overlay/b25a25abdd58e1df07bec3232d26112fd8c8b22ddbf41845c974c1ceefce4cf1/merged major:0 minor:405 fsType:overlay blockSize:0} overlay_0-407:{mountpoint:/var/lib/containers/storage/overlay/29403017faa74d14248575242da861de1b3a4f6674f32af557b48811c3e40436/merged major:0 minor:407 fsType:overlay blockSize:0} overlay_0-409:{mountpoint:/var/lib/containers/storage/overlay/dd0924e679223968cda0b85445ba7acc368b1a68539bfd8abc74ac0bee1a6caa/merged major:0 minor:409 fsType:overlay blockSize:0} overlay_0-411:{mountpoint:/var/lib/containers/storage/overlay/2982b1dc1be7b152e92739f76baad0af0c5cc2ba468ca0a2983f84a9a4ce4504/merged major:0 minor:411 fsType:overlay blockSize:0} overlay_0-413:{mountpoint:/var/lib/containers/storage/overlay/bd0c93e47bd2731c4de01737472e6b0d6e32140bc00681348f79e8b3c466c2ea/merged major:0 minor:413 fsType:overlay blockSize:0} overlay_0-427:{mountpoint:/var/lib/containers/storage/overlay/cc7574106cb6e45835c52c6b6c67c7f705dc61c77f1b44ed8e6d634215369aa7/merged major:0 minor:427 fsType:overlay blockSize:0} overlay_0-429:{mountpoint:/var/lib/containers/storage/overlay/452fba23d9db140b136c9647d151e0f62ee5e9689085acd09a2f284124fd24bc/merged major:0 minor:429 fsType:overlay blockSize:0} overlay_0-431:{mountpoint:/var/lib/containers/storage/overlay/bd1a6e555d5c04509498d9f669fc8a6e5a58752168b1c3bd6aaaa90f0af2d004/merged major:0 minor:431 fsType:overlay blockSize:0} overlay_0-437:{mountpoint:/var/lib/containers/storage/overlay/b2768225e185724bee7256cd1e0629bfa2b21736414ebc6e1612b136ed9f296e/merged major:0 minor:437 fsType:overlay blockSize:0} overlay_0-443:{mountpoint:/var/lib/containers/storage/overlay/c2fc6d7d73b0ac79b6b85511a4f56669e6066fa012ebbe76b341aaa192ca3b08/merged major:0 minor:443 fsType:overlay blockSize:0} overlay_0-445:{mountpoint:/var/lib/containers/storage/overlay/fe798286185be1dd01c686a6906ef6718bb39b75f5d1e8874364b30fc5e996d5/merged major:0 minor:445 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/cfde58d481bd384275967fadddd7f9da60a7bbf6a9ebd69c42f3dcbc0c2d30da/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/9eaf7f48e13284ae2e15fe0b3dcd83deafe1bd3adf9bb702bcaaece5403b05c3/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-71:{mountpoint:/var/lib/containers/storage/overlay/5909559a76e6e9299b25a8c5c103d1caf714d33ba602a7f17fe6de83a0b112d8/merged major:0 minor:71 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/5ec79a02f667f2b461b0f08331b7860b256c577d1b759110b90c1ee754159e39/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/f0301730846701e8b3d34c30f920a594faa0fa1189d50b77d272018ef73194aa/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/3ccd37f42b2f784fead08ee4a28ba5d21e31ba5150ab8374c79261e07566d7c7/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/3a42e2cdb1b7ebbf0c0ae7b2ecb3c7e5415591292287af685a5480e93d9614d5/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/7f5d3bd855a4efaf0da44017e36457bd083c8f7f6bb4426059287a1a61f232a6/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/c5ac691670bb7f3115f78d786b9d1d263c84cf0be59d2a58c84c988e4c1b377a/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/854cbbd7f98ead896e5c48f120c74634fe59fb798c959dc3945da1d9de65ff57/merged major:0 minor:96 fsType:overlay blockSize:0}]
Oct 30 21:50:22.461883 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.461473    5678 manager.go:217] Machine: {Timestamp:2026-10-30 21:50:22.460548631 +0000 UTC m=+0.064856186 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3092440 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2af5d5f1ca4eee0c8afb3964f4c493 SystemUUID:ec2af5d5-f1ca-4eee-0c8a-fb3964f4c493 BootID:a4655aac-a25f-4893-8d2e-73f95ef98b31 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f938a93b-29dd-4e7d-b8e9-444df68acf03/volumes/kubernetes.io~projected/kube-api-access-6jt5t DeviceMajor:0 DeviceMinor:286 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a3459ebb-2fc2-4a82-b9c0-de5542d667e1/volumes/kubernetes.io~projected/kube-api-access-6hdbq DeviceMajor:0 DeviceMinor:338 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-351 DeviceMajor:0 DeviceMinor:351 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-358 DeviceMajor:0 DeviceMinor:358 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ef7ae922-6da7-4817-ba91-35e67b6ac8c5/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:384 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ffcebcbe6a69d7a5b1acce836ecf6509f9155e88b6bd05c7b2f2dc9c025ae664/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/49ff8cb6-54de-4e8c-9a35-634f9a1ef29b/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:204 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/fdf54bb9-f735-4810-82ee-d803099a1bcf/volumes/kubernetes.io~projected/kube-api-access-s5jdv DeviceMajor:0 DeviceMinor:424 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ff7b45b007ad888b3545c4a198046fb18b188e5a1e0e7854aa6c5a60dc784808/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-356 DeviceMajor:0 DeviceMinor:356 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0ad59239-edcc-46a4-a5a1-1debb63e3a5d/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:58 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/516573f9-4ada-4868-b885-f61b9640e6bf/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-218 DeviceMajor:0 DeviceMinor:218 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-341 DeviceMajor:0 DeviceMinor:341 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d947d3db-ab6e-4874-8817-f8c74be138a1/volumes/kubernetes.io~projected/kube-api-access-mkkmr DeviceMajor:0 DeviceMinor:61 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f81df74784eeabd8abcee1495fda7eb479bb06f0826ce1b438d3632d3bd104b4/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/516573f9-4ada-4868-b885-f61b9640e6bf/volumes/kubernetes.io~projected/kube-api-access-66wt8 DeviceMajor:0 DeviceMinor:197 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5138fe8c-9927-4d05-8fe5-5556072ed595/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:303 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5138fe8c-9927-4d05-8fe5-5556072ed595/volume-subpaths/cluster-tls-config/alertmanager/10 DeviceMajor:0 DeviceMinor:295 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ef7ae922-6da7-4817-ba91-35e67b6ac8c5/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:377 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-427 DeviceMajor:0 DeviceMinor:427 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/d947d3db-ab6e-4874-8817-f8c74be138a1/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:55 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/82f75dd5-40cd-4cb3-9111-6213d6b43679/volumes/kubernetes.io~projected/kube-api-access-b97qt DeviceMajor:0 DeviceMinor:63 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5138fe8c-9927-4d05-8fe5-5556072ed595/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy DeviceMajor:0 DeviceMinor:301 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-330 DeviceMajor:0 DeviceMinor:330 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-443 DeviceMajor:0 DeviceMinor:443 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-445 DeviceMajor:0 DeviceMinor:445 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-232 DeviceMajor:0 DeviceMinor:232 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5138fe8c-9927-4d05-8fe5-5556072ed595/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:300 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5138fe8c-9927-4d05-8fe5-5556072ed595/volumes/kubernetes.io~projected/kube-api-access-6v9rf DeviceMajor:0 DeviceMinor:304 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ef7ae922-6da7-4817-ba91-35e67b6ac8c5/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:388 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5138fe8c-9927-4d05-8fe5-5556072ed595/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:299 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a7fc187ce460fbaeec60b28f3c313837ba07a8341b8639946b2382e42dde1919/userdata/shm DeviceMajor:0 DeviceMinor:339 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d96925669f74a34a6a8e0245a59131261ec4d0a7c88335b9eced7a1f596f1b04/userdata/shm DeviceMajor:0 DeviceMinor:403 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/929ab3025f8a48c7f0741da506a9447ead41f7e897a54256d70954ade722c613/userdata/shm DeviceMajor:0 DeviceMinor:212 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-255 DeviceMajor:0 DeviceMinor:255 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ef7ae922-6da7-4817-ba91-35e67b6ac8c5/volume-subpaths/web-config/config-reloader/2 DeviceMajor:0 DeviceMinor:377 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/da9e0ba8a0e86180f7612fe2a827ab57b9c0afcdf1a087ac9140a2b5de077636/userdata/shm DeviceMajor:0 DeviceMinor:208 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-216 DeviceMajor:0 DeviceMinor:216 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/99e016196628ee432af4e398621a5760c9b2739cc2b5a123742aa8523e334b05/userdata/shm DeviceMajor:0 DeviceMinor:287 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-343 DeviceMajor:0 DeviceMinor:343 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ef7ae922-6da7-4817-ba91-35e67b6ac8c5/volumes/kubernetes.io~secret/secret-kube-rbac-proxy DeviceMajor:0 DeviceMinor:386 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a6980f40-89cf-4f3d-8cd5-91969ae5a276/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:395 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-396 DeviceMajor:0 DeviceMinor:396 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-239 DeviceMajor:0 DeviceMinor:239 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4768f1ea299aaff0c7dc0db8d94ec8e94771f4682b4b37ea448a32a4eaa05704/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5138fe8c-9927-4d05-8fe5-5556072ed595/volumes/kubernetes.io~secret/cluster-tls-config DeviceMajor:0 DeviceMinor:295 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-411 DeviceMajor:0 DeviceMinor:411 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/67b50e2a-4d01-4d5f-a5d1-7e80ee62642a/volumes/kubernetes.io~secret/prometheus-operator-tls DeviceMajor:0 DeviceMinor:237 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-272 DeviceMajor:0 DeviceMinor:272 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/dec1cd54-fec6-4c12-9d37-a1b0b1a7ffc0/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:282 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-305 DeviceMajor:0 DeviceMinor:305 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5138fe8c-9927-4d05-8fe5-5556072ed595/volume-subpaths/web-config/config-reloader/6 DeviceMajor:0 DeviceMinor:299 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/fdf54bb9-f735-4810-82ee-d803099a1bcf/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:423 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/41fe6d63-9316-4916-b042-87ba7e602167/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/69e218ec-ab5e-4e09-8444-e53a710c53bf/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:203 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-227 DeviceMajor:0 DeviceMinor:227 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/67b50e2a-4d01-4d5f-a5d1-7e80ee62642a/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:238 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5138fe8c-9927-4d05-8fe5-5556072ed595/volume-subpaths/web-config/init-config-reloader/6 DeviceMajor:0 DeviceMinor:299 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-111 DeviceMajor:0 DeviceMinor:111 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ca41b3353f4cff5c25e5091dd9731c19f4a1577f6aceaf86e9b91db22ba7f5e1/userdata/shm DeviceMajor:0 DeviceMinor:214 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-229 DeviceMajor:0 DeviceMinor:229 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a3459ebb-2fc2-4a82-b9c0-de5542d667e1/volumes/kubernetes.io~secret/client-ca-bundle DeviceMajor:0 DeviceMinor:337 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ef7ae922-6da7-4817-ba91-35e67b6ac8c5/volumes/kubernetes.io~secret/secret-metrics-client-certs DeviceMajor:0 DeviceMinor:383 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/5f468f3c-bf3d-437c-8ec8-a5d56f448d99/volumes/kubernetes.io~projected/kube-api-access-rgn7x DeviceMajor:0 DeviceMinor:62 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a6980f40-89cf-4f3d-8cd5-91969ae5a276/volumes/kubernetes.io~projected/kube-api-access-czfxj DeviceMajor:0 DeviceMinor:66 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/49ff8cb6-54de-4e8c-9a35-634f9a1ef29b/volumes/kubernetes.io~projected/kube-api-access-z7qxb DeviceMajor:0 DeviceMinor:207 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-320 DeviceMajor:0 DeviceMinor:320 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d947d3db-ab6e-4874-8817-f8c74be138a1/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:56 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-220 DeviceMajor:0 DeviceMinor:220 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-222 DeviceMajor:0 DeviceMinor:222 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/dc24a46b097bebb5444228144f254835785f13e6c1da310ba365dff351a254f6/userdata/shm DeviceMajor:0 DeviceMinor:317 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7a6305de-8fd1-4d3e-9628-8c4cda07f379/volumes/kubernetes.io~projected/kube-api-access-lwvcm DeviceMajor:0 DeviceMinor:59 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-261 DeviceMajor:0 DeviceMinor:261 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f938a93b-29dd-4e7d-b8e9-444df68acf03/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:285 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b80c4122abd5858b56886f4d518ab4c81848d68f3a7fd253c2ebc4930ac56b10/userdata/shm DeviceMajor:0 DeviceMinor:289 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5138fe8c-9927-4d05-8fe5-5556072ed595/volumes/kubernetes.io~secret/config-volume DeviceMajor:0 DeviceMinor:298 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/41fe6d63-9316-4916-b042-87ba7e602167/volumes/kubernetes.io~projected/kube-api-access-2hghq DeviceMajor:0 DeviceMinor:60 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-291 DeviceMajor:0 DeviceMinor:291 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-324 DeviceMajor:0 DeviceMinor:324 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5138fe8c-9927-4d05-8fe5-5556072ed595/volume-subpaths/web-config/alertmanager/9 DeviceMajor:0 DeviceMinor:299 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-359 DeviceMajor:0 DeviceMinor:359 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ef7ae922-6da7-4817-ba91-35e67b6ac8c5/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls DeviceMajor:0 DeviceMinor:381 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5b7fa7c8959b27a9ea4f5af1c51cb119eefb9044902ab5949a2f83aa030ae7cb/userdata/shm DeviceMajor:0 DeviceMinor:77 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/48c8198c2f2b74b17130ae9678885f3f7d4c981c4bc55458dc71962684f9b0b8/userdata/shm DeviceMajor:0 DeviceMinor:93 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/67b50e2a-4d01-4d5f-a5d1-7e80ee62642a/volumes/kubernetes.io~projected/kube-api-access-fn582 DeviceMajor:0 DeviceMinor:244 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b954bab2-9dd7-4b39-bda6-3fec87381aff/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:394 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-405 DeviceMajor:0 DeviceMinor:405 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b958496c5215e685901b3b9da29749f1fd2fec924a1885b5175591646903879d/userdata/shm DeviceMajor:0 DeviceMinor:425 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/87c44883-54d0-4a05-b94b-d3c2835a03e1/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:201 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b8fd19e59bf3f63ce8b861120a1c5a46871e91627d90f3725696a99fc1e6f4f5/userdata/shm DeviceMajor:0 DeviceMinor:210 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-293 DeviceMajor:0 DeviceMinor:293 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-345 DeviceMajor:0 DeviceMinor:345 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-142 DeviceMajor:0 DeviceMinor:142 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/87c44883-54d0-4a05-b94b-d3c2835a03e1/volumes/kubernetes.io~projected/kube-api-access-cdznq DeviceMajor:0 DeviceMinor:206 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5138fe8c-9927-4d05-8fe5-5556072ed595/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric DeviceMajor:0 DeviceMinor:297 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2d4d5b332a32797e444f7b86043649ef9e8cd85554203c183f4263d812176bb9/userdata/shm DeviceMajor:0 DeviceMinor:401 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/run/containers/storage/overlay-containers/06de6ffcc0e4da71a6468cd1841f7fb4b2018aedf613cdaed926b55ac3d8b6fe/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-313 DeviceMajor:0 DeviceMinor:313 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ef7ae922-6da7-4817-ba91-35e67b6ac8c5/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file DeviceMajor:0 DeviceMinor:382 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/266940a224ea0a54c1a8a2d9b4e5b33f7b149ab8f92cc7b373d4259ede742beb/userdata/shm DeviceMajor:0 DeviceMinor:392 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3bd252982df6be8fd26cdcd2a455bb58da4c3b32f8bbbdc0df4c8a46ca2826fb/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-253 DeviceMajor:0 DeviceMinor:253 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a3459ebb-2fc2-4a82-b9c0-de5542d667e1/volumes/kubernetes.io~secret/secret-metrics-server-tls DeviceMajor:0 DeviceMinor:335 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/fdf54bb9-f735-4810-82ee-d803099a1bcf/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:419 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-431 DeviceMajor:0 DeviceMinor:431 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/41fe6d63-9316-4916-b042-87ba7e602167/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:57 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/69e218ec-ab5e-4e09-8444-e53a710c53bf/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:202 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/69e218ec-ab5e-4e09-8444-e53a710c53bf/volumes/kubernetes.io~projected/kube-api-access-gv6wf DeviceMajor:0 DeviceMinor:205 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ef7ae922-6da7-4817-ba91-35e67b6ac8c5/volumes/kubernetes.io~projected/kube-api-access-cr8x2 DeviceMajor:0 DeviceMinor:391 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/55619a2e-f6e9-4719-b362-30b69d553967/volumes/kubernetes.io~projected/kube-api-access-j6x28 DeviceMajor:0 DeviceMinor:400 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/54feac0378fd0894bff57a7d55fef92cb31ed8f5b2fcfe974fa7dcb286576b70/userdata/shm DeviceMajor:0 DeviceMinor:398 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/dec1cd54-fec6-4c12-9d37-a1b0b1a7ffc0/volumes/kubernetes.io~projected/kube-api-access-wvbdq DeviceMajor:0 DeviceMinor:283 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-307 DeviceMajor:0 DeviceMinor:307 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-413 DeviceMajor:0 DeviceMinor:413 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-437 DeviceMajor:0 DeviceMinor:437 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f6a186b19f8e7b056b1e01d0d6141ef6ea402c1dbb9a2d946a89669c6b60f63f/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-407 DeviceMajor:0 DeviceMinor:407 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-409 DeviceMajor:0 DeviceMinor:409 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/11dac6cc6798646aadb4d4e9dc28993e017dd18a9f1a636bf8789dce74026edd/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-372 DeviceMajor:0 DeviceMinor:372 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ef7ae922-6da7-4817-ba91-35e67b6ac8c5/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:390 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-429 DeviceMajor:0 DeviceMinor:429 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ef7ae922-6da7-4817-ba91-35e67b6ac8c5/volume-subpaths/web-config/prometheus/15 DeviceMajor:0 DeviceMinor:377 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ef9db6f7956a084b994e446966ee810675c93075b4589ea22626894079d461b1/userdata/shm DeviceMajor:0 DeviceMinor:70 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-71 DeviceMajor:0 DeviceMinor:71 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/32d63b18-0fcb-4e4a-a24c-ee044dcb5a90/volumes/kubernetes.io~projected/kube-api-access-pxd2f DeviceMajor:0 DeviceMinor:65 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/dec1cd54-fec6-4c12-9d37-a1b0b1a7ffc0/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:278 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ef7ae922-6da7-4817-ba91-35e67b6ac8c5/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls DeviceMajor:0 DeviceMinor:387 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ef7ae922-6da7-4817-ba91-35e67b6ac8c5/volumes/kubernetes.io~secret/config DeviceMajor:0 DeviceMinor:389 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5138fe8c-9927-4d05-8fe5-5556072ed595/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:302 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/924394f7-5b3b-4767-a1ff-f1d310ece4ed/volumes/kubernetes.io~projected/kube-api-access-qnt2l DeviceMajor:0 DeviceMinor:64 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f22e4514-ac6d-481d-8afe-6e2e4cdc3aa1/volumes/kubernetes.io~projected/kube-api-access-8w8vm DeviceMajor:0 DeviceMinor:67 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f938a93b-29dd-4e7d-b8e9-444df68acf03/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:284 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ef7ae922-6da7-4817-ba91-35e67b6ac8c5/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:385 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5138fe8c-9927-4d05-8fe5-5556072ed595/volumes/kubernetes.io~secret/secret-alertmanager-main-tls DeviceMajor:0 DeviceMinor:296 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a3459ebb-2fc2-4a82-b9c0-de5542d667e1/volumes/kubernetes.io~secret/secret-metrics-server-client-certs DeviceMajor:0 DeviceMinor:336 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ef7ae922-6da7-4817-ba91-35e67b6ac8c5/volume-subpaths/web-config/init-config-reloader/2 DeviceMajor:0 DeviceMinor:377 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:266940a224ea0a5 MacAddress:5a:32:4c:49:f1:4f Speed:10000 Mtu:8901} {Name:2d4d5b332a32797 MacAddress:e2:07:77:79:73:2c Speed:10000 Mtu:8901} {Name:4768f1ea299aaff MacAddress:f6:24:27:d3:f2:fe Speed:10000 Mtu:8901} {Name:54feac0378fd089 MacAddress:2e:fc:73:44:f0:89 Speed:10000 Mtu:8901} {Name:929ab3025f8a48c MacAddress:1a:c4:82:83:56:4e Speed:10000 Mtu:8901} {Name:99e016196628ee4 MacAddress:46:a0:7c:97:95:63 Speed:10000 Mtu:8901} {Name:a7fc187ce460fba MacAddress:12:d4:37:a6:85:12 Speed:10000 Mtu:8901} {Name:b8fd19e59bf3f63 MacAddress:8a:92:49:90:28:61 Speed:10000 Mtu:8901} {Name:b958496c5215e68 MacAddress:42:1a:aa:c0:61:da Speed:10000 Mtu:8901} {Name:br-ex MacAddress:06:ff:cf:e5:4f:8b Speed:0 Mtu:9001} {Name:br-int MacAddress:6a:30:62:a5:3f:7a Speed:0 Mtu:8901} {Name:ca41b3353f4cff5 MacAddress:3a:d2:1a:d3:13:0e Speed:10000 Mtu:8901} {Name:d96925669f74a34 MacAddress:7e:61:2d:50:2b:d0 Speed:10000 Mtu:8901} {Name:da9e0ba8a0e8618 MacAddress:ea:10:b1:3a:7a:e6 Speed:10000 Mtu:8901} {Name:dc24a46b097bebb MacAddress:d2:56:7c:cf:53:ff Speed:10000 Mtu:8901} {Name:ens5 MacAddress:06:ff:cf:e5:4f:8b Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:26:76:5a:2b:d0:44 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:8e:c3:ef:e6:a6:76 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 21:50:22.461883 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.461884    5678 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 21:50:22.462191 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.461930    5678 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 21:50:22.462228 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.462209    5678 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 21:50:22.462369 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.462243    5678 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-245.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 21:50:22.462438 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.462371    5678 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 21:50:22.462438 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.462379    5678 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 21:50:22.462438 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.462390    5678 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 21:50:22.462438 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.462426    5678 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 21:50:22.462548 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.462504    5678 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 21:50:22.462670 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.462660    5678 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 21:50:22.462772 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.462764    5678 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 21:50:22.462801 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.462798    5678 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 21:50:22.462828 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.462811    5678 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 21:50:22.462828 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.462821    5678 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 21:50:22.462894 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.462829    5678 kubelet.go:405] "Adding apiserver pod source"
Oct 30 21:50:22.462894 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.462837    5678 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 21:50:22.463463 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.463450    5678 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 21:50:22.463463 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.463463    5678 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 21:50:22.463990 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.463973    5678 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 21:50:22.475879 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.475864    5678 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 21:50:22.475879 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.475878    5678 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 21:50:22.475997 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.475979    5678 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 30 21:50:22.476168 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.476156    5678 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 21:50:22.476205 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.476185    5678 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 21:50:22.476280 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.476271    5678 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 21:50:22.476311 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.476285    5678 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 21:50:22.476311 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.476291    5678 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 21:50:22.476311 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.476297    5678 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 21:50:22.476311 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.476303    5678 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 21:50:22.476311 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.476308    5678 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 21:50:22.476472 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.476313    5678 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 21:50:22.476472 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.476319    5678 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 21:50:22.476472 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.476326    5678 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 21:50:22.476472 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.476332    5678 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 21:50:22.476472 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.476349    5678 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 21:50:22.476472 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.476361    5678 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 21:50:22.476472 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.476391    5678 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 21:50:22.476472 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.476411    5678 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 21:50:22.479771 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.479757    5678 server.go:1263] "Started kubelet"
Oct 30 21:50:22.479855 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.479826    5678 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 21:50:22.479918 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.479859    5678 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 21:50:22.479975 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.479955    5678 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 21:50:22.480305 ip-10-0-128-245 systemd[1]: Started Kubernetes Kubelet.
Oct 30 21:50:22.481654 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.480789    5678 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 21:50:22.481783 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.481765    5678 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 21:50:22.487011 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.486991    5678 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 21:50:22.487011 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.487007    5678 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 21:50:22.487185 ip-10-0-128-245 kubenswrapper[5678]: E1030 21:50:22.487098    5678 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-245.us-west-2.compute.internal\" not found"
Oct 30 21:50:22.487185 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.487110    5678 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 21:50:22.487185 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.487152    5678 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 21:50:22.487341 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.487118    5678 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 21:50:22.489483 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.489465    5678 factory.go:55] Registering systemd factory
Oct 30 21:50:22.489483 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.489484    5678 factory.go:223] Registration of the systemd container factory successfully
Oct 30 21:50:22.489700 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.489687    5678 factory.go:153] Registering CRI-O factory
Oct 30 21:50:22.489761 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.489702    5678 factory.go:223] Registration of the crio container factory successfully
Oct 30 21:50:22.489761 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.489751    5678 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 21:50:22.489863 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.489769    5678 factory.go:103] Registering Raw factory
Oct 30 21:50:22.489863 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.489782    5678 manager.go:1199] Started watching for new ooms in manager
Oct 30 21:50:22.490064 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.490036    5678 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="32d63b18-0fcb-4e4a-a24c-ee044dcb5a90" volumeName="kubernetes.io/projected/32d63b18-0fcb-4e4a-a24c-ee044dcb5a90-kube-api-access-pxd2f" seLinuxMountContext=""
Oct 30 21:50:22.490168 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.490063    5678 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="41fe6d63-9316-4916-b042-87ba7e602167" volumeName="kubernetes.io/configmap/41fe6d63-9316-4916-b042-87ba7e602167-ovnkube-config" seLinuxMountContext=""
Oct 30 21:50:22.490168 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.490073    5678 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="49ff8cb6-54de-4e8c-9a35-634f9a1ef29b" volumeName="kubernetes.io/projected/49ff8cb6-54de-4e8c-9a35-634f9a1ef29b-kube-api-access-z7qxb" seLinuxMountContext=""
Oct 30 21:50:22.490168 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.490083    5678 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5138fe8c-9927-4d05-8fe5-5556072ed595" volumeName="kubernetes.io/empty-dir/5138fe8c-9927-4d05-8fe5-5556072ed595-alertmanager-main-db" seLinuxMountContext=""
Oct 30 21:50:22.490168 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.490095    5678 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5f468f3c-bf3d-437c-8ec8-a5d56f448d99" volumeName="kubernetes.io/configmap/5f468f3c-bf3d-437c-8ec8-a5d56f448d99-multus-daemon-config" seLinuxMountContext=""
Oct 30 21:50:22.490168 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.490103    5678 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a3459ebb-2fc2-4a82-b9c0-de5542d667e1" volumeName="kubernetes.io/secret/a3459ebb-2fc2-4a82-b9c0-de5542d667e1-secret-metrics-server-tls" seLinuxMountContext=""
Oct 30 21:50:22.490168 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.490111    5678 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a6980f40-89cf-4f3d-8cd5-91969ae5a276" volumeName="kubernetes.io/projected/a6980f40-89cf-4f3d-8cd5-91969ae5a276-kube-api-access-czfxj" seLinuxMountContext=""
Oct 30 21:50:22.490168 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.490118    5678 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="41fe6d63-9316-4916-b042-87ba7e602167" volumeName="kubernetes.io/configmap/41fe6d63-9316-4916-b042-87ba7e602167-env-overrides" seLinuxMountContext=""
Oct 30 21:50:22.490168 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.490127    5678 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="516573f9-4ada-4868-b885-f61b9640e6bf" volumeName="kubernetes.io/secret/516573f9-4ada-4868-b885-f61b9640e6bf-cert" seLinuxMountContext=""
Oct 30 21:50:22.490168 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.490141    5678 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7a6305de-8fd1-4d3e-9628-8c4cda07f379" volumeName="kubernetes.io/projected/7a6305de-8fd1-4d3e-9628-8c4cda07f379-kube-api-access-lwvcm" seLinuxMountContext=""
Oct 30 21:50:22.490168 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.490148    5678 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d947d3db-ab6e-4874-8817-f8c74be138a1" volumeName="kubernetes.io/empty-dir/d947d3db-ab6e-4874-8817-f8c74be138a1-tmp" seLinuxMountContext=""
Oct 30 21:50:22.490168 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.490157    5678 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d947d3db-ab6e-4874-8817-f8c74be138a1" volumeName="kubernetes.io/projected/d947d3db-ab6e-4874-8817-f8c74be138a1-kube-api-access-mkkmr" seLinuxMountContext=""
Oct 30 21:50:22.490168 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.490170    5678 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ef7ae922-6da7-4817-ba91-35e67b6ac8c5" volumeName="kubernetes.io/configmap/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-prometheus-k8s-rulefiles-1" seLinuxMountContext=""
Oct 30 21:50:22.490788 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.490179    5678 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ef7ae922-6da7-4817-ba91-35e67b6ac8c5" volumeName="kubernetes.io/configmap/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-configmap-serving-certs-ca-bundle" seLinuxMountContext=""
Oct 30 21:50:22.490788 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.490192    5678 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ef7ae922-6da7-4817-ba91-35e67b6ac8c5" volumeName="kubernetes.io/empty-dir/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-config-out" seLinuxMountContext=""
Oct 30 21:50:22.490788 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.490199    5678 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="69e218ec-ab5e-4e09-8444-e53a710c53bf" volumeName="kubernetes.io/projected/69e218ec-ab5e-4e09-8444-e53a710c53bf-kube-api-access-gv6wf" seLinuxMountContext=""
Oct 30 21:50:22.490788 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.490215    5678 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="87c44883-54d0-4a05-b94b-d3c2835a03e1" volumeName="kubernetes.io/projected/87c44883-54d0-4a05-b94b-d3c2835a03e1-kube-api-access-cdznq" seLinuxMountContext=""
Oct 30 21:50:22.490788 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.490223    5678 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d947d3db-ab6e-4874-8817-f8c74be138a1" volumeName="kubernetes.io/empty-dir/d947d3db-ab6e-4874-8817-f8c74be138a1-etc-tuned" seLinuxMountContext=""
Oct 30 21:50:22.490788 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.490232    5678 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dec1cd54-fec6-4c12-9d37-a1b0b1a7ffc0" volumeName="kubernetes.io/empty-dir/dec1cd54-fec6-4c12-9d37-a1b0b1a7ffc0-volume-directive-shadow" seLinuxMountContext=""
Oct 30 21:50:22.490788 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.490264    5678 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dec1cd54-fec6-4c12-9d37-a1b0b1a7ffc0" volumeName="kubernetes.io/secret/dec1cd54-fec6-4c12-9d37-a1b0b1a7ffc0-kube-state-metrics-tls" seLinuxMountContext=""
Oct 30 21:50:22.490788 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.490275    5678 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f938a93b-29dd-4e7d-b8e9-444df68acf03" volumeName="kubernetes.io/configmap/f938a93b-29dd-4e7d-b8e9-444df68acf03-node-exporter-accelerators-collector-config" seLinuxMountContext=""
Oct 30 21:50:22.490788 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.490283    5678 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f938a93b-29dd-4e7d-b8e9-444df68acf03" volumeName="kubernetes.io/secret/f938a93b-29dd-4e7d-b8e9-444df68acf03-node-exporter-tls" seLinuxMountContext=""
Oct 30 21:50:22.490788 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.490290    5678 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fdf54bb9-f735-4810-82ee-d803099a1bcf" volumeName="kubernetes.io/configmap/fdf54bb9-f735-4810-82ee-d803099a1bcf-service-ca" seLinuxMountContext=""
Oct 30 21:50:22.490788 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.490297    5678 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="67b50e2a-4d01-4d5f-a5d1-7e80ee62642a" volumeName="kubernetes.io/secret/67b50e2a-4d01-4d5f-a5d1-7e80ee62642a-prometheus-operator-kube-rbac-proxy-config" seLinuxMountContext=""
Oct 30 21:50:22.490788 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.490305    5678 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7a6305de-8fd1-4d3e-9628-8c4cda07f379" volumeName="kubernetes.io/configmap/7a6305de-8fd1-4d3e-9628-8c4cda07f379-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 30 21:50:22.490788 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.490313    5678 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ef7ae922-6da7-4817-ba91-35e67b6ac8c5" volumeName="kubernetes.io/projected/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-tls-assets" seLinuxMountContext=""
Oct 30 21:50:22.490788 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.490320    5678 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ef7ae922-6da7-4817-ba91-35e67b6ac8c5" volumeName="kubernetes.io/projected/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-kube-api-access-cr8x2" seLinuxMountContext=""
Oct 30 21:50:22.490788 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.490328    5678 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ef7ae922-6da7-4817-ba91-35e67b6ac8c5" volumeName="kubernetes.io/secret/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-secret-prometheus-k8s-tls" seLinuxMountContext=""
Oct 30 21:50:22.490788 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.490345    5678 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ef7ae922-6da7-4817-ba91-35e67b6ac8c5" volumeName="kubernetes.io/secret/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-secret-kube-rbac-proxy" seLinuxMountContext=""
Oct 30 21:50:22.490788 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.490353    5678 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fdf54bb9-f735-4810-82ee-d803099a1bcf" volumeName="kubernetes.io/empty-dir/fdf54bb9-f735-4810-82ee-d803099a1bcf-tmp" seLinuxMountContext=""
Oct 30 21:50:22.490788 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.490360    5678 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fdf54bb9-f735-4810-82ee-d803099a1bcf" volumeName="kubernetes.io/projected/fdf54bb9-f735-4810-82ee-d803099a1bcf-kube-api-access-s5jdv" seLinuxMountContext=""
Oct 30 21:50:22.490788 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.490367    5678 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="41fe6d63-9316-4916-b042-87ba7e602167" volumeName="kubernetes.io/configmap/41fe6d63-9316-4916-b042-87ba7e602167-ovnkube-script-lib" seLinuxMountContext=""
Oct 30 21:50:22.490788 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.490375    5678 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="41fe6d63-9316-4916-b042-87ba7e602167" volumeName="kubernetes.io/projected/41fe6d63-9316-4916-b042-87ba7e602167-kube-api-access-2hghq" seLinuxMountContext=""
Oct 30 21:50:22.490788 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.490384    5678 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5138fe8c-9927-4d05-8fe5-5556072ed595" volumeName="kubernetes.io/projected/5138fe8c-9927-4d05-8fe5-5556072ed595-tls-assets" seLinuxMountContext=""
Oct 30 21:50:22.490788 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.490391    5678 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5f468f3c-bf3d-437c-8ec8-a5d56f448d99" volumeName="kubernetes.io/projected/5f468f3c-bf3d-437c-8ec8-a5d56f448d99-kube-api-access-rgn7x" seLinuxMountContext=""
Oct 30 21:50:22.490788 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.490416    5678 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="87c44883-54d0-4a05-b94b-d3c2835a03e1" volumeName="kubernetes.io/configmap/87c44883-54d0-4a05-b94b-d3c2835a03e1-kube-rbac-proxy-cm" seLinuxMountContext=""
Oct 30 21:50:22.490788 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.490428    5678 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a6980f40-89cf-4f3d-8cd5-91969ae5a276" volumeName="kubernetes.io/secret/a6980f40-89cf-4f3d-8cd5-91969ae5a276-metrics-certs" seLinuxMountContext=""
Oct 30 21:50:22.490788 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.490439    5678 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dec1cd54-fec6-4c12-9d37-a1b0b1a7ffc0" volumeName="kubernetes.io/secret/dec1cd54-fec6-4c12-9d37-a1b0b1a7ffc0-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext=""
Oct 30 21:50:22.490788 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.490450    5678 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ef7ae922-6da7-4817-ba91-35e67b6ac8c5" volumeName="kubernetes.io/secret/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-config" seLinuxMountContext=""
Oct 30 21:50:22.490788 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.490461    5678 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5138fe8c-9927-4d05-8fe5-5556072ed595" volumeName="kubernetes.io/secret/5138fe8c-9927-4d05-8fe5-5556072ed595-secret-alertmanager-main-tls" seLinuxMountContext=""
Oct 30 21:50:22.490788 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.490473    5678 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5138fe8c-9927-4d05-8fe5-5556072ed595" volumeName="kubernetes.io/secret/5138fe8c-9927-4d05-8fe5-5556072ed595-web-config" seLinuxMountContext=""
Oct 30 21:50:22.490788 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.490500    5678 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="67b50e2a-4d01-4d5f-a5d1-7e80ee62642a" volumeName="kubernetes.io/projected/67b50e2a-4d01-4d5f-a5d1-7e80ee62642a-kube-api-access-fn582" seLinuxMountContext=""
Oct 30 21:50:22.490788 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.490514    5678 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a3459ebb-2fc2-4a82-b9c0-de5542d667e1" volumeName="kubernetes.io/configmap/a3459ebb-2fc2-4a82-b9c0-de5542d667e1-configmap-kubelet-serving-ca-bundle" seLinuxMountContext=""
Oct 30 21:50:22.490788 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.490526    5678 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a3459ebb-2fc2-4a82-b9c0-de5542d667e1" volumeName="kubernetes.io/secret/a3459ebb-2fc2-4a82-b9c0-de5542d667e1-secret-metrics-server-client-certs" seLinuxMountContext=""
Oct 30 21:50:22.490788 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.490540    5678 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f22e4514-ac6d-481d-8afe-6e2e4cdc3aa1" volumeName="kubernetes.io/configmap/f22e4514-ac6d-481d-8afe-6e2e4cdc3aa1-iptables-alerter-script" seLinuxMountContext=""
Oct 30 21:50:22.490788 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.490554    5678 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fdf54bb9-f735-4810-82ee-d803099a1bcf" volumeName="kubernetes.io/configmap/fdf54bb9-f735-4810-82ee-d803099a1bcf-console-config" seLinuxMountContext=""
Oct 30 21:50:22.490788 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.490568    5678 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5138fe8c-9927-4d05-8fe5-5556072ed595" volumeName="kubernetes.io/configmap/5138fe8c-9927-4d05-8fe5-5556072ed595-metrics-client-ca" seLinuxMountContext=""
Oct 30 21:50:22.490788 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.490584    5678 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5138fe8c-9927-4d05-8fe5-5556072ed595" volumeName="kubernetes.io/projected/5138fe8c-9927-4d05-8fe5-5556072ed595-kube-api-access-6v9rf" seLinuxMountContext=""
Oct 30 21:50:22.490788 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.490616    5678 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a3459ebb-2fc2-4a82-b9c0-de5542d667e1" volumeName="kubernetes.io/secret/a3459ebb-2fc2-4a82-b9c0-de5542d667e1-client-ca-bundle" seLinuxMountContext=""
Oct 30 21:50:22.490788 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.490628    5678 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ef7ae922-6da7-4817-ba91-35e67b6ac8c5" volumeName="kubernetes.io/secret/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-web-config" seLinuxMountContext=""
Oct 30 21:50:22.490788 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.490075    5678 manager.go:319] Starting recovery of all containers
Oct 30 21:50:22.490788 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.490639    5678 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f938a93b-29dd-4e7d-b8e9-444df68acf03" volumeName="kubernetes.io/empty-dir/f938a93b-29dd-4e7d-b8e9-444df68acf03-node-exporter-textfile" seLinuxMountContext=""
Oct 30 21:50:22.490788 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.490709    5678 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fdf54bb9-f735-4810-82ee-d803099a1bcf" volumeName="kubernetes.io/configmap/fdf54bb9-f735-4810-82ee-d803099a1bcf-trusted-ca-bundle" seLinuxMountContext=""
Oct 30 21:50:22.490788 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.490724    5678 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="49ff8cb6-54de-4e8c-9a35-634f9a1ef29b" volumeName="kubernetes.io/empty-dir/49ff8cb6-54de-4e8c-9a35-634f9a1ef29b-tmp-dir" seLinuxMountContext=""
Oct 30 21:50:22.490788 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.490736    5678 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="67b50e2a-4d01-4d5f-a5d1-7e80ee62642a" volumeName="kubernetes.io/configmap/67b50e2a-4d01-4d5f-a5d1-7e80ee62642a-metrics-client-ca" seLinuxMountContext=""
Oct 30 21:50:22.490788 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.490760    5678 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="87c44883-54d0-4a05-b94b-d3c2835a03e1" volumeName="kubernetes.io/secret/87c44883-54d0-4a05-b94b-d3c2835a03e1-insights-runtime-extractor-tls" seLinuxMountContext=""
Oct 30 21:50:22.490788 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.490772    5678 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dec1cd54-fec6-4c12-9d37-a1b0b1a7ffc0" volumeName="kubernetes.io/projected/dec1cd54-fec6-4c12-9d37-a1b0b1a7ffc0-kube-api-access-wvbdq" seLinuxMountContext=""
Oct 30 21:50:22.490788 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.490795    5678 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="82f75dd5-40cd-4cb3-9111-6213d6b43679" volumeName="kubernetes.io/projected/82f75dd5-40cd-4cb3-9111-6213d6b43679-kube-api-access-b97qt" seLinuxMountContext=""
Oct 30 21:50:22.492832 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.490840    5678 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5138fe8c-9927-4d05-8fe5-5556072ed595" volumeName="kubernetes.io/empty-dir/5138fe8c-9927-4d05-8fe5-5556072ed595-config-out" seLinuxMountContext=""
Oct 30 21:50:22.492832 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.490854    5678 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="516573f9-4ada-4868-b885-f61b9640e6bf" volumeName="kubernetes.io/projected/516573f9-4ada-4868-b885-f61b9640e6bf-kube-api-access-66wt8" seLinuxMountContext=""
Oct 30 21:50:22.492832 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.490865    5678 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="69e218ec-ab5e-4e09-8444-e53a710c53bf" volumeName="kubernetes.io/configmap/69e218ec-ab5e-4e09-8444-e53a710c53bf-console-config" seLinuxMountContext=""
Oct 30 21:50:22.492832 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.490880    5678 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="69e218ec-ab5e-4e09-8444-e53a710c53bf" volumeName="kubernetes.io/configmap/69e218ec-ab5e-4e09-8444-e53a710c53bf-service-ca" seLinuxMountContext=""
Oct 30 21:50:22.492832 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.490925    5678 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b954bab2-9dd7-4b39-bda6-3fec87381aff" volumeName="kubernetes.io/secret/b954bab2-9dd7-4b39-bda6-3fec87381aff-original-pull-secret" seLinuxMountContext=""
Oct 30 21:50:22.492832 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.490937    5678 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ef7ae922-6da7-4817-ba91-35e67b6ac8c5" volumeName="kubernetes.io/configmap/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-prometheus-trusted-ca-bundle" seLinuxMountContext=""
Oct 30 21:50:22.492832 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.490960    5678 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="32d63b18-0fcb-4e4a-a24c-ee044dcb5a90" volumeName="kubernetes.io/empty-dir/32d63b18-0fcb-4e4a-a24c-ee044dcb5a90-tmp-dir" seLinuxMountContext=""
Oct 30 21:50:22.492832 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.490972    5678 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="49ff8cb6-54de-4e8c-9a35-634f9a1ef29b" volumeName="kubernetes.io/secret/49ff8cb6-54de-4e8c-9a35-634f9a1ef29b-metrics-tls" seLinuxMountContext=""
Oct 30 21:50:22.492832 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.490993    5678 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5138fe8c-9927-4d05-8fe5-5556072ed595" volumeName="kubernetes.io/secret/5138fe8c-9927-4d05-8fe5-5556072ed595-config-volume" seLinuxMountContext=""
Oct 30 21:50:22.492832 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.491005    5678 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="55619a2e-f6e9-4719-b362-30b69d553967" volumeName="kubernetes.io/projected/55619a2e-f6e9-4719-b362-30b69d553967-kube-api-access-j6x28" seLinuxMountContext=""
Oct 30 21:50:22.492832 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.491018    5678 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5f468f3c-bf3d-437c-8ec8-a5d56f448d99" volumeName="kubernetes.io/configmap/5f468f3c-bf3d-437c-8ec8-a5d56f448d99-cni-binary-copy" seLinuxMountContext=""
Oct 30 21:50:22.492832 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.491031    5678 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ef7ae922-6da7-4817-ba91-35e67b6ac8c5" volumeName="kubernetes.io/configmap/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-prometheus-k8s-rulefiles-2" seLinuxMountContext=""
Oct 30 21:50:22.492832 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.491053    5678 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="82f75dd5-40cd-4cb3-9111-6213d6b43679" volumeName="kubernetes.io/configmap/82f75dd5-40cd-4cb3-9111-6213d6b43679-serviceca" seLinuxMountContext=""
Oct 30 21:50:22.492832 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.491072    5678 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ef7ae922-6da7-4817-ba91-35e67b6ac8c5" volumeName="kubernetes.io/configmap/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-configmap-metrics-client-ca" seLinuxMountContext=""
Oct 30 21:50:22.492832 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.491084    5678 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ef7ae922-6da7-4817-ba91-35e67b6ac8c5" volumeName="kubernetes.io/secret/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-secret-prometheus-k8s-thanos-sidecar-tls" seLinuxMountContext=""
Oct 30 21:50:22.492832 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.491099    5678 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f938a93b-29dd-4e7d-b8e9-444df68acf03" volumeName="kubernetes.io/configmap/f938a93b-29dd-4e7d-b8e9-444df68acf03-metrics-client-ca" seLinuxMountContext=""
Oct 30 21:50:22.492832 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.491147    5678 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f938a93b-29dd-4e7d-b8e9-444df68acf03" volumeName="kubernetes.io/projected/f938a93b-29dd-4e7d-b8e9-444df68acf03-kube-api-access-6jt5t" seLinuxMountContext=""
Oct 30 21:50:22.492832 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.491160    5678 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fdf54bb9-f735-4810-82ee-d803099a1bcf" volumeName="kubernetes.io/secret/fdf54bb9-f735-4810-82ee-d803099a1bcf-console-oauth-config" seLinuxMountContext=""
Oct 30 21:50:22.492832 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.491173    5678 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7a6305de-8fd1-4d3e-9628-8c4cda07f379" volumeName="kubernetes.io/configmap/7a6305de-8fd1-4d3e-9628-8c4cda07f379-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 30 21:50:22.492832 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.491184    5678 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="41fe6d63-9316-4916-b042-87ba7e602167" volumeName="kubernetes.io/secret/41fe6d63-9316-4916-b042-87ba7e602167-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 30 21:50:22.492832 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.491195    5678 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5138fe8c-9927-4d05-8fe5-5556072ed595" volumeName="kubernetes.io/secret/5138fe8c-9927-4d05-8fe5-5556072ed595-cluster-tls-config" seLinuxMountContext=""
Oct 30 21:50:22.492832 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.491216    5678 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5138fe8c-9927-4d05-8fe5-5556072ed595" volumeName="kubernetes.io/secret/5138fe8c-9927-4d05-8fe5-5556072ed595-secret-alertmanager-kube-rbac-proxy-metric" seLinuxMountContext=""
Oct 30 21:50:22.492832 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.491227    5678 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="69e218ec-ab5e-4e09-8444-e53a710c53bf" volumeName="kubernetes.io/configmap/69e218ec-ab5e-4e09-8444-e53a710c53bf-oauth-serving-cert" seLinuxMountContext=""
Oct 30 21:50:22.492832 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.491250    5678 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="69e218ec-ab5e-4e09-8444-e53a710c53bf" volumeName="kubernetes.io/empty-dir/69e218ec-ab5e-4e09-8444-e53a710c53bf-tmp" seLinuxMountContext=""
Oct 30 21:50:22.492832 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.491302    5678 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a3459ebb-2fc2-4a82-b9c0-de5542d667e1" volumeName="kubernetes.io/empty-dir/a3459ebb-2fc2-4a82-b9c0-de5542d667e1-audit-log" seLinuxMountContext=""
Oct 30 21:50:22.492832 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.491314    5678 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ef7ae922-6da7-4817-ba91-35e67b6ac8c5" volumeName="kubernetes.io/empty-dir/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-prometheus-k8s-db" seLinuxMountContext=""
Oct 30 21:50:22.492832 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.491329    5678 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5138fe8c-9927-4d05-8fe5-5556072ed595" volumeName="kubernetes.io/configmap/5138fe8c-9927-4d05-8fe5-5556072ed595-alertmanager-trusted-ca-bundle" seLinuxMountContext=""
Oct 30 21:50:22.492832 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.491362    5678 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5138fe8c-9927-4d05-8fe5-5556072ed595" volumeName="kubernetes.io/secret/5138fe8c-9927-4d05-8fe5-5556072ed595-secret-alertmanager-kube-rbac-proxy-web" seLinuxMountContext=""
Oct 30 21:50:22.492832 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.491375    5678 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dec1cd54-fec6-4c12-9d37-a1b0b1a7ffc0" volumeName="kubernetes.io/configmap/dec1cd54-fec6-4c12-9d37-a1b0b1a7ffc0-metrics-client-ca" seLinuxMountContext=""
Oct 30 21:50:22.492832 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.491386    5678 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ef7ae922-6da7-4817-ba91-35e67b6ac8c5" volumeName="kubernetes.io/configmap/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-configmap-kubelet-serving-ca-bundle" seLinuxMountContext=""
Oct 30 21:50:22.492832 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.491412    5678 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ef7ae922-6da7-4817-ba91-35e67b6ac8c5" volumeName="kubernetes.io/configmap/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-prometheus-k8s-rulefiles-0" seLinuxMountContext=""
Oct 30 21:50:22.492832 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.491424    5678 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ef7ae922-6da7-4817-ba91-35e67b6ac8c5" volumeName="kubernetes.io/secret/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-secret-metrics-client-certs" seLinuxMountContext=""
Oct 30 21:50:22.492832 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.491448    5678 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f22e4514-ac6d-481d-8afe-6e2e4cdc3aa1" volumeName="kubernetes.io/projected/f22e4514-ac6d-481d-8afe-6e2e4cdc3aa1-kube-api-access-8w8vm" seLinuxMountContext=""
Oct 30 21:50:22.492832 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.491460    5678 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f938a93b-29dd-4e7d-b8e9-444df68acf03" volumeName="kubernetes.io/secret/f938a93b-29dd-4e7d-b8e9-444df68acf03-node-exporter-kube-rbac-proxy-config" seLinuxMountContext=""
Oct 30 21:50:22.492832 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.491471    5678 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0ad59239-edcc-46a4-a5a1-1debb63e3a5d" volumeName="kubernetes.io/secret/0ad59239-edcc-46a4-a5a1-1debb63e3a5d-agent-certs" seLinuxMountContext=""
Oct 30 21:50:22.492832 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.491482    5678 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="49ff8cb6-54de-4e8c-9a35-634f9a1ef29b" volumeName="kubernetes.io/configmap/49ff8cb6-54de-4e8c-9a35-634f9a1ef29b-config-volume" seLinuxMountContext=""
Oct 30 21:50:22.492832 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.491493    5678 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="67b50e2a-4d01-4d5f-a5d1-7e80ee62642a" volumeName="kubernetes.io/secret/67b50e2a-4d01-4d5f-a5d1-7e80ee62642a-prometheus-operator-tls" seLinuxMountContext=""
Oct 30 21:50:22.492832 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.491506    5678 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="87c44883-54d0-4a05-b94b-d3c2835a03e1" volumeName="kubernetes.io/empty-dir/87c44883-54d0-4a05-b94b-d3c2835a03e1-data-volume" seLinuxMountContext=""
Oct 30 21:50:22.492832 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.491517    5678 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a3459ebb-2fc2-4a82-b9c0-de5542d667e1" volumeName="kubernetes.io/projected/a3459ebb-2fc2-4a82-b9c0-de5542d667e1-kube-api-access-6hdbq" seLinuxMountContext=""
Oct 30 21:50:22.492832 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.491528    5678 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ef7ae922-6da7-4817-ba91-35e67b6ac8c5" volumeName="kubernetes.io/secret/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-thanos-prometheus-http-client-file" seLinuxMountContext=""
Oct 30 21:50:22.492832 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.491539    5678 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ef7ae922-6da7-4817-ba91-35e67b6ac8c5" volumeName="kubernetes.io/secret/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-secret-grpc-tls" seLinuxMountContext=""
Oct 30 21:50:22.492832 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.491550    5678 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5138fe8c-9927-4d05-8fe5-5556072ed595" volumeName="kubernetes.io/secret/5138fe8c-9927-4d05-8fe5-5556072ed595-secret-alertmanager-kube-rbac-proxy" seLinuxMountContext=""
Oct 30 21:50:22.492832 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.491562    5678 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="69e218ec-ab5e-4e09-8444-e53a710c53bf" volumeName="kubernetes.io/secret/69e218ec-ab5e-4e09-8444-e53a710c53bf-console-oauth-config" seLinuxMountContext=""
Oct 30 21:50:22.492832 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.491573    5678 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="924394f7-5b3b-4767-a1ff-f1d310ece4ed" volumeName="kubernetes.io/projected/924394f7-5b3b-4767-a1ff-f1d310ece4ed-kube-api-access-qnt2l" seLinuxMountContext=""
Oct 30 21:50:22.492832 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.491585    5678 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dec1cd54-fec6-4c12-9d37-a1b0b1a7ffc0" volumeName="kubernetes.io/configmap/dec1cd54-fec6-4c12-9d37-a1b0b1a7ffc0-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext=""
Oct 30 21:50:22.492832 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.491605    5678 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fdf54bb9-f735-4810-82ee-d803099a1bcf" volumeName="kubernetes.io/secret/fdf54bb9-f735-4810-82ee-d803099a1bcf-console-serving-cert" seLinuxMountContext=""
Oct 30 21:50:22.492832 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.491619    5678 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="69e218ec-ab5e-4e09-8444-e53a710c53bf" volumeName="kubernetes.io/secret/69e218ec-ab5e-4e09-8444-e53a710c53bf-console-serving-cert" seLinuxMountContext=""
Oct 30 21:50:22.492832 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.491630    5678 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7a6305de-8fd1-4d3e-9628-8c4cda07f379" volumeName="kubernetes.io/configmap/7a6305de-8fd1-4d3e-9628-8c4cda07f379-cni-binary-copy" seLinuxMountContext=""
Oct 30 21:50:22.492832 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.491676    5678 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a3459ebb-2fc2-4a82-b9c0-de5542d667e1" volumeName="kubernetes.io/configmap/a3459ebb-2fc2-4a82-b9c0-de5542d667e1-metrics-server-audit-profiles" seLinuxMountContext=""
Oct 30 21:50:22.492832 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.491687    5678 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ef7ae922-6da7-4817-ba91-35e67b6ac8c5" volumeName="kubernetes.io/secret/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-secret-prometheus-k8s-kube-rbac-proxy-web" seLinuxMountContext=""
Oct 30 21:50:22.492832 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.491698    5678 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fdf54bb9-f735-4810-82ee-d803099a1bcf" volumeName="kubernetes.io/configmap/fdf54bb9-f735-4810-82ee-d803099a1bcf-oauth-serving-cert" seLinuxMountContext=""
Oct 30 21:50:22.492832 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.491718    5678 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0ad59239-edcc-46a4-a5a1-1debb63e3a5d" volumeName="kubernetes.io/configmap/0ad59239-edcc-46a4-a5a1-1debb63e3a5d-konnectivity-ca" seLinuxMountContext=""
Oct 30 21:50:22.492832 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.491729    5678 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 21:50:22.492832 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.491737    5678 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 21:50:22.492832 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.487042    5678 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 21:44:11 +0000 UTC" deadline="2028-05-24 01:45:54.343237955 +0000 UTC"
Oct 30 21:50:22.492832 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.492451    5678 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13707h55m31.850793294s"
Oct 30 21:50:22.513542 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.513493    5678 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 21:50:22.514995 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.514975    5678 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 21:50:22.514995 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.514994    5678 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 21:50:22.515144 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.515013    5678 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 21:50:22.515144 ip-10-0-128-245 kubenswrapper[5678]: E1030 21:50:22.515060    5678 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 30 21:50:22.522011 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.521969    5678 generic.go:356] "Generic (PLEG): container finished" podID="f938a93b-29dd-4e7d-b8e9-444df68acf03" containerID="a2140afa979bca517055031d1bee6eddcc47ed7fcc16f3bba2d16f3da48c99d7" exitCode=0
Oct 30 21:50:22.540714 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.540690    5678 generic.go:356] "Generic (PLEG): container finished" podID="7a6305de-8fd1-4d3e-9628-8c4cda07f379" containerID="2a80cb2c3c33e04cec680df1a093e34cab4c2a5ae1862fad0dff04b2e211caef" exitCode=0
Oct 30 21:50:22.540714 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.540713    5678 generic.go:356] "Generic (PLEG): container finished" podID="7a6305de-8fd1-4d3e-9628-8c4cda07f379" containerID="3c1c44d1979b97f91d92a71a5f56aed61bf71890981234dc6b289cb0763da087" exitCode=0
Oct 30 21:50:22.540940 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.540720    5678 generic.go:356] "Generic (PLEG): container finished" podID="7a6305de-8fd1-4d3e-9628-8c4cda07f379" containerID="82a62e13298910f03ade52d27ed8119eadace52fd94746cb80d6dc8beddbabd0" exitCode=0
Oct 30 21:50:22.540940 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.540726    5678 generic.go:356] "Generic (PLEG): container finished" podID="7a6305de-8fd1-4d3e-9628-8c4cda07f379" containerID="9d4a8af82f0954e2903c897898b8de663da1eca550d940952fe5f1253cb6d2b2" exitCode=0
Oct 30 21:50:22.540940 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.540732    5678 generic.go:356] "Generic (PLEG): container finished" podID="7a6305de-8fd1-4d3e-9628-8c4cda07f379" containerID="f349a356942ed99fcc929e902dc50e6cee3c4a40e4efe6d1f46a6d0a549d61a7" exitCode=0
Oct 30 21:50:22.540940 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.540738    5678 generic.go:356] "Generic (PLEG): container finished" podID="7a6305de-8fd1-4d3e-9628-8c4cda07f379" containerID="5e17da19f23daf9a0c51c74fd19a0094ab93c0ee4d6f7dbe83e2847177a17fad" exitCode=0
Oct 30 21:50:22.570177 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.570147    5678 generic.go:356] "Generic (PLEG): container finished" podID="ef7ae922-6da7-4817-ba91-35e67b6ac8c5" containerID="258fd66deee3797d90e92c40203fe9ebf6228e43b399cf1d0332b8034deed533" exitCode=0
Oct 30 21:50:22.573658 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.573633    5678 generic.go:356] "Generic (PLEG): container finished" podID="5138fe8c-9927-4d05-8fe5-5556072ed595" containerID="f7f271fecf3370b4e1a7117367120bc8e1a3ba6129d0a2171e8c002bee61190a" exitCode=0
Oct 30 21:50:22.577157 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.577088    5678 generic.go:356] "Generic (PLEG): container finished" podID="d8412ee5d9c7152e66cb6933cbd6fbc8" containerID="72424879b5ac678cd40f9fd655a0a39b1bea72c1ff1d689e32996870e99d35ea" exitCode=0
Oct 30 21:50:22.587215 ip-10-0-128-245 kubenswrapper[5678]: E1030 21:50:22.587193    5678 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-245.us-west-2.compute.internal\" not found"
Oct 30 21:50:22.596274 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.596250    5678 manager.go:324] Recovery completed
Oct 30 21:50:22.615126 ip-10-0-128-245 kubenswrapper[5678]: E1030 21:50:22.615101    5678 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Oct 30 21:50:22.617147 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.617130    5678 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:50:22.617450 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.617433    5678 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:50:22.619207 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.619193    5678 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-245.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:50:22.619273 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.619220    5678 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-245.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:50:22.619273 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.619230    5678 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-245.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:50:22.621842 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.621827    5678 cpu_manager.go:226] "Starting" policy="none"
Oct 30 21:50:22.621842 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.621840    5678 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 21:50:22.621939 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.621846    5678 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:50:22.621939 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.621856    5678 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 21:50:22.621993 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.621976    5678 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 30 21:50:22.622033 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.621987    5678 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 30 21:50:22.622033 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.622004    5678 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 30 21:50:22.622033 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.622010    5678 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 30 21:50:22.622033 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.622015    5678 policy_none.go:50] "Start"
Oct 30 21:50:22.622033 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.622024    5678 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 21:50:22.622175 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.622036    5678 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 21:50:22.622175 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.622167    5678 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 30 21:50:22.622175 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.622174    5678 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 30 21:50:22.622250 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.622180    5678 policy_none.go:44] "Start"
Oct 30 21:50:22.622654 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.622636    5678 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-245.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:50:22.622736 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.622663    5678 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-245.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:50:22.622736 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.622675    5678 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-245.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:50:22.626769 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.626746    5678 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 21:50:22.629318 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.629307    5678 manager.go:341] "Starting Device Plugin manager"
Oct 30 21:50:22.629363 ip-10-0-128-245 kubenswrapper[5678]: E1030 21:50:22.629328    5678 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 21:50:22.629363 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.629337    5678 server.go:87] "Starting device plugin registration server"
Oct 30 21:50:22.629618 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.629606    5678 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 21:50:22.629674 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.629618    5678 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 21:50:22.629757 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.629734    5678 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 21:50:22.629882 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.629844    5678 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 21:50:22.629946 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.629893    5678 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 21:50:22.629946 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.629901    5678 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 21:50:22.630511 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.629966    5678 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T21:50:22.629881242Z","UUID":"6d6ce222-4009-4579-9e6e-ccd5add5aaec","Handler":null,"Name":"","Endpoint":""}
Oct 30 21:50:22.631596 ip-10-0-128-245 kubenswrapper[5678]: E1030 21:50:22.631574    5678 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 21:50:22.631663 ip-10-0-128-245 kubenswrapper[5678]: E1030 21:50:22.631618    5678 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-245.us-west-2.compute.internal\" not found"
Oct 30 21:50:22.631707 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.631683    5678 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 21:50:22.631707 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.631704    5678 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 21:50:22.662949 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.662932    5678 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 30 21:50:22.730368 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.730344    5678 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:50:22.731332 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.731312    5678 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-245.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:50:22.731439 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.731339    5678 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-245.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:50:22.731439 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.731348    5678 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-245.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:50:22.731439 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.731413    5678 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-128-245.us-west-2.compute.internal"
Oct 30 21:50:22.754984 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.754956    5678 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:50:22.755080 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.755058    5678 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:50:22.756700 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.756685    5678 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:50:22.815772 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.815722    5678 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-245.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-245.us-west-2.compute.internal"]
Oct 30 21:50:22.816433 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.816372    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-245.us-west-2.compute.internal" event={"ID":"bb28c8c054a6af006e24e10c141990f3","Type":"ContainerStarted","Data":"c531e24ed710fdc8c273ee2562bad7c804ee7684d3ed9abe8fbc89b9eb3227e6"}
Oct 30 21:50:22.816548 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.816447    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-245.us-west-2.compute.internal" event={"ID":"bb28c8c054a6af006e24e10c141990f3","Type":"ContainerStarted","Data":"3bd252982df6be8fd26cdcd2a455bb58da4c3b32f8bbbdc0df4c8a46ca2826fb"}
Oct 30 21:50:22.816548 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.816484    5678 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-245.us-west-2.compute.internal"
Oct 30 21:50:22.816548 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.816505    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-245.us-west-2.compute.internal" event={"ID":"d8412ee5d9c7152e66cb6933cbd6fbc8","Type":"ContainerStarted","Data":"b6c09ab85e7691eed67138e66f9d1c5bd9f3bd17c0d1a4acfaf523cbb0a5f06a"}
Oct 30 21:50:22.816548 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.816524    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-245.us-west-2.compute.internal" event={"ID":"d8412ee5d9c7152e66cb6933cbd6fbc8","Type":"ContainerDied","Data":"72424879b5ac678cd40f9fd655a0a39b1bea72c1ff1d689e32996870e99d35ea"}
Oct 30 21:50:22.816548 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.816542    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-245.us-west-2.compute.internal" event={"ID":"d8412ee5d9c7152e66cb6933cbd6fbc8","Type":"ContainerStarted","Data":"ffcebcbe6a69d7a5b1acce836ecf6509f9155e88b6bd05c7b2f2dc9c025ae664"}
Oct 30 21:50:22.816729 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.816486    5678 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-245.us-west-2.compute.internal"
Oct 30 21:50:22.835564 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.835513    5678 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-128-245.us-west-2.compute.internal"
Oct 30 21:50:22.835654 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.835568    5678 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-128-245.us-west-2.compute.internal"
Oct 30 21:50:22.874567 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.874547    5678 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 21:50:22.874658 ip-10-0-128-245 kubenswrapper[5678]: E1030 21:50:22.874582    5678 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-128-245.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-245.us-west-2.compute.internal"
Oct 30 21:50:22.875042 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.875030    5678 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 21:50:22.875075 ip-10-0-128-245 kubenswrapper[5678]: E1030 21:50:22.875058    5678 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-128-245.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-245.us-west-2.compute.internal"
Oct 30 21:50:22.964167 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.964137    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d8412ee5d9c7152e66cb6933cbd6fbc8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-245.us-west-2.compute.internal\" (UID: \"d8412ee5d9c7152e66cb6933cbd6fbc8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-245.us-west-2.compute.internal"
Oct 30 21:50:22.964364 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.964173    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d8412ee5d9c7152e66cb6933cbd6fbc8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-245.us-west-2.compute.internal\" (UID: \"d8412ee5d9c7152e66cb6933cbd6fbc8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-245.us-west-2.compute.internal"
Oct 30 21:50:22.964364 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:22.964263    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bb28c8c054a6af006e24e10c141990f3-config\") pod \"kube-apiserver-proxy-ip-10-0-128-245.us-west-2.compute.internal\" (UID: \"bb28c8c054a6af006e24e10c141990f3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-245.us-west-2.compute.internal"
Oct 30 21:50:23.064877 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.064841    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d8412ee5d9c7152e66cb6933cbd6fbc8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-245.us-west-2.compute.internal\" (UID: \"d8412ee5d9c7152e66cb6933cbd6fbc8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-245.us-west-2.compute.internal"
Oct 30 21:50:23.065019 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.064888    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d8412ee5d9c7152e66cb6933cbd6fbc8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-245.us-west-2.compute.internal\" (UID: \"d8412ee5d9c7152e66cb6933cbd6fbc8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-245.us-west-2.compute.internal"
Oct 30 21:50:23.065019 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.064929    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bb28c8c054a6af006e24e10c141990f3-config\") pod \"kube-apiserver-proxy-ip-10-0-128-245.us-west-2.compute.internal\" (UID: \"bb28c8c054a6af006e24e10c141990f3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-245.us-west-2.compute.internal"
Oct 30 21:50:23.065019 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.064965    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d8412ee5d9c7152e66cb6933cbd6fbc8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-245.us-west-2.compute.internal\" (UID: \"d8412ee5d9c7152e66cb6933cbd6fbc8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-245.us-west-2.compute.internal"
Oct 30 21:50:23.065145 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.064962    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d8412ee5d9c7152e66cb6933cbd6fbc8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-245.us-west-2.compute.internal\" (UID: \"d8412ee5d9c7152e66cb6933cbd6fbc8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-245.us-west-2.compute.internal"
Oct 30 21:50:23.065145 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.064978    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bb28c8c054a6af006e24e10c141990f3-config\") pod \"kube-apiserver-proxy-ip-10-0-128-245.us-west-2.compute.internal\" (UID: \"bb28c8c054a6af006e24e10c141990f3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-245.us-west-2.compute.internal"
Oct 30 21:50:23.463720 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.463694    5678 apiserver.go:52] "Watching apiserver"
Oct 30 21:50:23.562543 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.562511    5678 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 21:50:23.563205 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.563168    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-7cbdbcf4b5-rmbsz","openshift-network-operator/iptables-alerter-t24lz","openshift-console/console-69874d4cdd-zhlp5","openshift-ingress-canary/ingress-canary-hr227","openshift-multus/multus-p7zpx","openshift-ovn-kubernetes/ovnkube-node-ml9xf","kube-system/global-pull-secret-syncer-cck7d","openshift-cluster-node-tuning-operator/tuned-22pl2","openshift-multus/multus-additional-cni-plugins-vrtnk","openshift-insights/insights-runtime-extractor-kn6xx","openshift-monitoring/kube-state-metrics-5dc8d7d89b-pvl2l","openshift-dns/node-resolver-gbpvx","kube-system/konnectivity-agent-w72c9","kube-system/kube-apiserver-proxy-ip-10-0-128-245.us-west-2.compute.internal","openshift-image-registry/node-ca-gmbbw","openshift-monitoring/prometheus-k8s-0","openshift-network-diagnostics/network-check-target-w9tx8","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-245.us-west-2.compute.internal","openshift-monitoring/metrics-server-dc7c7d47d-ghdn7","openshift-multus/network-metrics-daemon-xbh9m","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lrc2c","openshift-dns/dns-default-fdmdz","openshift-monitoring/alertmanager-main-0","openshift-monitoring/node-exporter-7j5j6","openshift-monitoring/prometheus-operator-9cd89b74c-vcj4q"]
Oct 30 21:50:23.584569 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.584543    5678 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-245.us-west-2.compute.internal"
Oct 30 21:50:23.584695 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.584665    5678 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-245.us-west-2.compute.internal"
Oct 30 21:50:23.589525 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.589509    5678 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 21:50:23.614084 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.614058    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 21:50:23.614203 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.614099    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-9mttd\""
Oct 30 21:50:23.614438 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.614422    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9bjlv\""
Oct 30 21:50:23.614923 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.614902    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 21:50:23.615130 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.614966    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 21:50:23.615130 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.615026    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 21:50:23.615130 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.615075    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 21:50:23.615497 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.615178    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 21:50:23.615497 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.615242    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 21:50:23.615497 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.615248    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 21:50:23.615497 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.615277    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 21:50:23.615497 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.615316    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 21:50:23.615497 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.615368    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-gbc4r\""
Oct 30 21:50:23.615497 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.615390    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 21:50:23.615497 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.615440    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 21:50:23.615497 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.615506    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-cmhxq\""
Oct 30 21:50:23.615497 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.615513    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 21:50:23.615497 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.615519    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-92jf8\""
Oct 30 21:50:23.615497 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.615531    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 21:50:23.616305 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.615565    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 21:50:23.616305 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.615643    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 21:50:23.616305 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.615677    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 21:50:23.616305 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.615680    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-jbflf\""
Oct 30 21:50:23.616305 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.615726    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-hm4gw\""
Oct 30 21:50:23.616305 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.615835    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 21:50:23.616305 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.616075    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 21:50:23.616305 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.616105    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 21:50:23.616305 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.616105    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 21:50:23.616305 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.616098    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-7d2ft\""
Oct 30 21:50:23.616305 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.616175    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 21:50:23.616305 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.616171    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 21:50:23.616305 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.616202    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 21:50:23.616305 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.616223    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-dvh7z\""
Oct 30 21:50:23.616305 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.616244    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 21:50:23.616856 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.616496    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-9mgc8\""
Oct 30 21:50:23.616856 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.616716    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-nwhdt\""
Oct 30 21:50:23.617047 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.617035    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 21:50:23.617091 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.617057    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 21:50:23.617202 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.617191    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 21:50:23.660573 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.660541    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 21:50:23.660818 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.660795    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Oct 30 21:50:23.660818 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.660800    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 21:50:23.660999 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.660895    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Oct 30 21:50:23.663001 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.662982    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Oct 30 21:50:23.664964 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.664946    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Oct 30 21:50:23.665075 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.664978    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 21:50:23.665158 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.665031    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 21:50:23.665233 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.665069    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 21:50:23.665233 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.665167    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 30 21:50:23.665319 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.665254    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Oct 30 21:50:23.665704 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.665684    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Oct 30 21:50:23.666112 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.666091    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 21:50:23.666210 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.666140    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 21:50:23.666210 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.666157    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 21:50:23.666210 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.666168    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Oct 30 21:50:23.666210 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.666186    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 21:50:23.666417 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.666217    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Oct 30 21:50:23.666417 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.666219    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Oct 30 21:50:23.666417 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.666235    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Oct 30 21:50:23.666417 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.666243    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 30 21:50:23.666417 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.666248    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 21:50:23.666417 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.666329    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Oct 30 21:50:23.666659 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.666540    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Oct 30 21:50:23.666659 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.666544    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Oct 30 21:50:23.666659 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.666554    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Oct 30 21:50:23.666659 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.666574    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Oct 30 21:50:23.666659 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.666591    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Oct 30 21:50:23.666659 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.666623    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Oct 30 21:50:23.666659 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.666645    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Oct 30 21:50:23.666955 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.666690    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Oct 30 21:50:23.666955 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.666699    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Oct 30 21:50:23.666955 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.666706    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Oct 30 21:50:23.666955 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.666707    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Oct 30 21:50:23.666955 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.666728    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Oct 30 21:50:23.666955 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.666738    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Oct 30 21:50:23.666955 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.666874    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Oct 30 21:50:23.666955 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.666920    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Oct 30 21:50:23.666955 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.666930    5678 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 21:50:23.666955 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.666956    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Oct 30 21:50:23.667436 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.666972    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Oct 30 21:50:23.667436 ip-10-0-128-245 kubenswrapper[5678]: E1030 21:50:23.666997    5678 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-128-245.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-245.us-west-2.compute.internal"
Oct 30 21:50:23.667436 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.667021    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-bbh9g\""
Oct 30 21:50:23.667436 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.667027    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 21:50:23.667436 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.667067    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 21:50:23.667436 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.667070    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 30 21:50:23.667436 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.667136    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-lxh58\""
Oct 30 21:50:23.667436 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.667199    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-4qz2z\""
Oct 30 21:50:23.667436 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.667299    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 30 21:50:23.667436 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.667318    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Oct 30 21:50:23.667436 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.667323    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-bmu48km85ucup\""
Oct 30 21:50:23.667436 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.667348    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-nzxnt\""
Oct 30 21:50:23.667436 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.667380    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-dhl5t\""
Oct 30 21:50:23.667436 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.667418    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Oct 30 21:50:23.667436 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.667436    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-jlg2v\""
Oct 30 21:50:23.668064 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.667467    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 21:50:23.668064 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.667521    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-52nbt\""
Oct 30 21:50:23.668064 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.667735    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-6gwn9\""
Oct 30 21:50:23.668064 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.667739    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 21:50:23.668064 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.667773    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-gnnmn\""
Oct 30 21:50:23.668064 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.667781    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Oct 30 21:50:23.668064 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.667862    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Oct 30 21:50:23.668064 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.667878    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Oct 30 21:50:23.668064 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.667928    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Oct 30 21:50:23.668064 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.667973    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Oct 30 21:50:23.668496 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.668294    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Oct 30 21:50:23.668496 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.668375    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-8lg5r\""
Oct 30 21:50:23.668496 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.668479    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-kfpck\""
Oct 30 21:50:23.668735 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.668710    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/dec1cd54-fec6-4c12-9d37-a1b0b1a7ffc0-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-pvl2l\" (UID: \"dec1cd54-fec6-4c12-9d37-a1b0b1a7ffc0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pvl2l"
Oct 30 21:50:23.668796 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.668740    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/41fe6d63-9316-4916-b042-87ba7e602167-host-cni-netd\") pod \"ovnkube-node-ml9xf\" (UID: \"41fe6d63-9316-4916-b042-87ba7e602167\") " pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf"
Oct 30 21:50:23.668796 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.668766    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/41fe6d63-9316-4916-b042-87ba7e602167-env-overrides\") pod \"ovnkube-node-ml9xf\" (UID: \"41fe6d63-9316-4916-b042-87ba7e602167\") " pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf"
Oct 30 21:50:23.668796 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.668787    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:23.668901 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.668811    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:23.668995 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.668973    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:23.669057 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.669015    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d947d3db-ab6e-4874-8817-f8c74be138a1-etc-systemd\") pod \"tuned-22pl2\" (UID: \"d947d3db-ab6e-4874-8817-f8c74be138a1\") " pod="openshift-cluster-node-tuning-operator/tuned-22pl2"
Oct 30 21:50:23.669057 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.669033    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/dec1cd54-fec6-4c12-9d37-a1b0b1a7ffc0-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-pvl2l\" (UID: \"dec1cd54-fec6-4c12-9d37-a1b0b1a7ffc0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pvl2l"
Oct 30 21:50:23.669057 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.669043    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fdf54bb9-f735-4810-82ee-d803099a1bcf-console-config\") pod \"console-7cbdbcf4b5-rmbsz\" (UID: \"fdf54bb9-f735-4810-82ee-d803099a1bcf\") " pod="openshift-console/console-7cbdbcf4b5-rmbsz"
Oct 30 21:50:23.669196 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.669065    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fdf54bb9-f735-4810-82ee-d803099a1bcf-tmp\") pod \"console-7cbdbcf4b5-rmbsz\" (UID: \"fdf54bb9-f735-4810-82ee-d803099a1bcf\") " pod="openshift-console/console-7cbdbcf4b5-rmbsz"
Oct 30 21:50:23.669196 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.669095    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5f468f3c-bf3d-437c-8ec8-a5d56f448d99-cni-binary-copy\") pod \"multus-p7zpx\" (UID: \"5f468f3c-bf3d-437c-8ec8-a5d56f448d99\") " pod="openshift-multus/multus-p7zpx"
Oct 30 21:50:23.669196 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.669100    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:23.669196 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.669120    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgn7x\" (UniqueName: \"kubernetes.io/projected/5f468f3c-bf3d-437c-8ec8-a5d56f448d99-kube-api-access-rgn7x\") pod \"multus-p7zpx\" (UID: \"5f468f3c-bf3d-437c-8ec8-a5d56f448d99\") " pod="openshift-multus/multus-p7zpx"
Oct 30 21:50:23.669196 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.669147    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f22e4514-ac6d-481d-8afe-6e2e4cdc3aa1-iptables-alerter-script\") pod \"iptables-alerter-t24lz\" (UID: \"f22e4514-ac6d-481d-8afe-6e2e4cdc3aa1\") " pod="openshift-network-operator/iptables-alerter-t24lz"
Oct 30 21:50:23.669196 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.669174    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/82f75dd5-40cd-4cb3-9111-6213d6b43679-serviceca\") pod \"node-ca-gmbbw\" (UID: \"82f75dd5-40cd-4cb3-9111-6213d6b43679\") " pod="openshift-image-registry/node-ca-gmbbw"
Oct 30 21:50:23.669196 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.669189    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:23.669536 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.669206    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7a6305de-8fd1-4d3e-9628-8c4cda07f379-cnibin\") pod \"multus-additional-cni-plugins-vrtnk\" (UID: \"7a6305de-8fd1-4d3e-9628-8c4cda07f379\") " pod="openshift-multus/multus-additional-cni-plugins-vrtnk"
Oct 30 21:50:23.669536 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.669234    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/41fe6d63-9316-4916-b042-87ba7e602167-env-overrides\") pod \"ovnkube-node-ml9xf\" (UID: \"41fe6d63-9316-4916-b042-87ba7e602167\") " pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf"
Oct 30 21:50:23.669536 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.669236    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5f468f3c-bf3d-437c-8ec8-a5d56f448d99-cnibin\") pod \"multus-p7zpx\" (UID: \"5f468f3c-bf3d-437c-8ec8-a5d56f448d99\") " pod="openshift-multus/multus-p7zpx"
Oct 30 21:50:23.669536 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.669309    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5f468f3c-bf3d-437c-8ec8-a5d56f448d99-host-run-multus-certs\") pod \"multus-p7zpx\" (UID: \"5f468f3c-bf3d-437c-8ec8-a5d56f448d99\") " pod="openshift-multus/multus-p7zpx"
Oct 30 21:50:23.669536 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.669326    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fdf54bb9-f735-4810-82ee-d803099a1bcf-console-config\") pod \"console-7cbdbcf4b5-rmbsz\" (UID: \"fdf54bb9-f735-4810-82ee-d803099a1bcf\") " pod="openshift-console/console-7cbdbcf4b5-rmbsz"
Oct 30 21:50:23.669536 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.669330    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/41fe6d63-9316-4916-b042-87ba7e602167-run-openvswitch\") pod \"ovnkube-node-ml9xf\" (UID: \"41fe6d63-9316-4916-b042-87ba7e602167\") " pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf"
Oct 30 21:50:23.669536 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.669352    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5138fe8c-9927-4d05-8fe5-5556072ed595-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5138fe8c-9927-4d05-8fe5-5556072ed595\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:50:23.669536 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.669376    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/924394f7-5b3b-4767-a1ff-f1d310ece4ed-device-dir\") pod \"aws-ebs-csi-driver-node-lrc2c\" (UID: \"924394f7-5b3b-4767-a1ff-f1d310ece4ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lrc2c"
Oct 30 21:50:23.670369 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.670345    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fdf54bb9-f735-4810-82ee-d803099a1bcf-tmp\") pod \"console-7cbdbcf4b5-rmbsz\" (UID: \"fdf54bb9-f735-4810-82ee-d803099a1bcf\") " pod="openshift-console/console-7cbdbcf4b5-rmbsz"
Oct 30 21:50:23.670574 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.670549    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f22e4514-ac6d-481d-8afe-6e2e4cdc3aa1-iptables-alerter-script\") pod \"iptables-alerter-t24lz\" (UID: \"f22e4514-ac6d-481d-8afe-6e2e4cdc3aa1\") " pod="openshift-network-operator/iptables-alerter-t24lz"
Oct 30 21:50:23.670756 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.670723    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5138fe8c-9927-4d05-8fe5-5556072ed595-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5138fe8c-9927-4d05-8fe5-5556072ed595\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:50:23.671425 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.671374    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/82f75dd5-40cd-4cb3-9111-6213d6b43679-serviceca\") pod \"node-ca-gmbbw\" (UID: \"82f75dd5-40cd-4cb3-9111-6213d6b43679\") " pod="openshift-image-registry/node-ca-gmbbw"
Oct 30 21:50:23.671753 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.671734    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:23.672010 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.671985    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5f468f3c-bf3d-437c-8ec8-a5d56f448d99-cni-binary-copy\") pod \"multus-p7zpx\" (UID: \"5f468f3c-bf3d-437c-8ec8-a5d56f448d99\") " pod="openshift-multus/multus-p7zpx"
Oct 30 21:50:23.673558 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.672334    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5f468f3c-bf3d-437c-8ec8-a5d56f448d99-multus-conf-dir\") pod \"multus-p7zpx\" (UID: \"5f468f3c-bf3d-437c-8ec8-a5d56f448d99\") " pod="openshift-multus/multus-p7zpx"
Oct 30 21:50:23.673558 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.672430    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f938a93b-29dd-4e7d-b8e9-444df68acf03-node-exporter-tls\") pod \"node-exporter-7j5j6\" (UID: \"f938a93b-29dd-4e7d-b8e9-444df68acf03\") " pod="openshift-monitoring/node-exporter-7j5j6"
Oct 30 21:50:23.673558 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.672533    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/67b50e2a-4d01-4d5f-a5d1-7e80ee62642a-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-vcj4q\" (UID: \"67b50e2a-4d01-4d5f-a5d1-7e80ee62642a\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-vcj4q"
Oct 30 21:50:23.673558 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.672667    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32d63b18-0fcb-4e4a-a24c-ee044dcb5a90-tmp-dir\") pod \"node-resolver-gbpvx\" (UID: \"32d63b18-0fcb-4e4a-a24c-ee044dcb5a90\") " pod="openshift-dns/node-resolver-gbpvx"
Oct 30 21:50:23.673558 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.672706    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:23.673558 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.672738    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:23.673558 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.672796    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d947d3db-ab6e-4874-8817-f8c74be138a1-run\") pod \"tuned-22pl2\" (UID: \"d947d3db-ab6e-4874-8817-f8c74be138a1\") " pod="openshift-cluster-node-tuning-operator/tuned-22pl2"
Oct 30 21:50:23.673558 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.672935    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5f468f3c-bf3d-437c-8ec8-a5d56f448d99-host-var-lib-cni-bin\") pod \"multus-p7zpx\" (UID: \"5f468f3c-bf3d-437c-8ec8-a5d56f448d99\") " pod="openshift-multus/multus-p7zpx"
Oct 30 21:50:23.673558 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.672975    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f938a93b-29dd-4e7d-b8e9-444df68acf03-node-exporter-wtmp\") pod \"node-exporter-7j5j6\" (UID: \"f938a93b-29dd-4e7d-b8e9-444df68acf03\") " pod="openshift-monitoring/node-exporter-7j5j6"
Oct 30 21:50:23.673558 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.673030    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/dec1cd54-fec6-4c12-9d37-a1b0b1a7ffc0-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-pvl2l\" (UID: \"dec1cd54-fec6-4c12-9d37-a1b0b1a7ffc0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pvl2l"
Oct 30 21:50:23.673558 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.673063    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fdf54bb9-f735-4810-82ee-d803099a1bcf-service-ca\") pod \"console-7cbdbcf4b5-rmbsz\" (UID: \"fdf54bb9-f735-4810-82ee-d803099a1bcf\") " pod="openshift-console/console-7cbdbcf4b5-rmbsz"
Oct 30 21:50:23.673558 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.673093    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5jdv\" (UniqueName: \"kubernetes.io/projected/fdf54bb9-f735-4810-82ee-d803099a1bcf-kube-api-access-s5jdv\") pod \"console-7cbdbcf4b5-rmbsz\" (UID: \"fdf54bb9-f735-4810-82ee-d803099a1bcf\") " pod="openshift-console/console-7cbdbcf4b5-rmbsz"
Oct 30 21:50:23.673558 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.673123    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5f468f3c-bf3d-437c-8ec8-a5d56f448d99-system-cni-dir\") pod \"multus-p7zpx\" (UID: \"5f468f3c-bf3d-437c-8ec8-a5d56f448d99\") " pod="openshift-multus/multus-p7zpx"
Oct 30 21:50:23.673558 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.673163    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d947d3db-ab6e-4874-8817-f8c74be138a1-etc-modprobe-d\") pod \"tuned-22pl2\" (UID: \"d947d3db-ab6e-4874-8817-f8c74be138a1\") " pod="openshift-cluster-node-tuning-operator/tuned-22pl2"
Oct 30 21:50:23.673558 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.673191    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d947d3db-ab6e-4874-8817-f8c74be138a1-etc-sysctl-d\") pod \"tuned-22pl2\" (UID: \"d947d3db-ab6e-4874-8817-f8c74be138a1\") " pod="openshift-cluster-node-tuning-operator/tuned-22pl2"
Oct 30 21:50:23.673558 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.673493    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5138fe8c-9927-4d05-8fe5-5556072ed595-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5138fe8c-9927-4d05-8fe5-5556072ed595\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:50:23.673558 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.673543    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5138fe8c-9927-4d05-8fe5-5556072ed595-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5138fe8c-9927-4d05-8fe5-5556072ed595\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:50:23.674030 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.673570    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5f468f3c-bf3d-437c-8ec8-a5d56f448d99-host-var-lib-cni-multus\") pod \"multus-p7zpx\" (UID: \"5f468f3c-bf3d-437c-8ec8-a5d56f448d99\") " pod="openshift-multus/multus-p7zpx"
Oct 30 21:50:23.674030 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.673608    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f938a93b-29dd-4e7d-b8e9-444df68acf03-sys\") pod \"node-exporter-7j5j6\" (UID: \"f938a93b-29dd-4e7d-b8e9-444df68acf03\") " pod="openshift-monitoring/node-exporter-7j5j6"
Oct 30 21:50:23.674030 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.673638    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/a3459ebb-2fc2-4a82-b9c0-de5542d667e1-secret-metrics-server-tls\") pod \"metrics-server-dc7c7d47d-ghdn7\" (UID: \"a3459ebb-2fc2-4a82-b9c0-de5542d667e1\") " pod="openshift-monitoring/metrics-server-dc7c7d47d-ghdn7"
Oct 30 21:50:23.674030 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.673680    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f938a93b-29dd-4e7d-b8e9-444df68acf03-node-exporter-accelerators-collector-config\") pod \"node-exporter-7j5j6\" (UID: \"f938a93b-29dd-4e7d-b8e9-444df68acf03\") " pod="openshift-monitoring/node-exporter-7j5j6"
Oct 30 21:50:23.674030 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.673720    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f22e4514-ac6d-481d-8afe-6e2e4cdc3aa1-host-slash\") pod \"iptables-alerter-t24lz\" (UID: \"f22e4514-ac6d-481d-8afe-6e2e4cdc3aa1\") " pod="openshift-network-operator/iptables-alerter-t24lz"
Oct 30 21:50:23.674030 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.673764    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7a6305de-8fd1-4d3e-9628-8c4cda07f379-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vrtnk\" (UID: \"7a6305de-8fd1-4d3e-9628-8c4cda07f379\") " pod="openshift-multus/multus-additional-cni-plugins-vrtnk"
Oct 30 21:50:23.674030 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.673837    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:23.674030 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.673907    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/69e218ec-ab5e-4e09-8444-e53a710c53bf-console-serving-cert\") pod \"console-69874d4cdd-zhlp5\" (UID: \"69e218ec-ab5e-4e09-8444-e53a710c53bf\") " pod="openshift-console/console-69874d4cdd-zhlp5"
Oct 30 21:50:23.674030 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.673937    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/69e218ec-ab5e-4e09-8444-e53a710c53bf-tmp\") pod \"console-69874d4cdd-zhlp5\" (UID: \"69e218ec-ab5e-4e09-8444-e53a710c53bf\") " pod="openshift-console/console-69874d4cdd-zhlp5"
Oct 30 21:50:23.674030 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.673965    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f938a93b-29dd-4e7d-b8e9-444df68acf03-node-exporter-textfile\") pod \"node-exporter-7j5j6\" (UID: \"f938a93b-29dd-4e7d-b8e9-444df68acf03\") " pod="openshift-monitoring/node-exporter-7j5j6"
Oct 30 21:50:23.674030 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.674006    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b97qt\" (UniqueName: \"kubernetes.io/projected/82f75dd5-40cd-4cb3-9111-6213d6b43679-kube-api-access-b97qt\") pod \"node-ca-gmbbw\" (UID: \"82f75dd5-40cd-4cb3-9111-6213d6b43679\") " pod="openshift-image-registry/node-ca-gmbbw"
Oct 30 21:50:23.674309 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.674050    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hghq\" (UniqueName: \"kubernetes.io/projected/41fe6d63-9316-4916-b042-87ba7e602167-kube-api-access-2hghq\") pod \"ovnkube-node-ml9xf\" (UID: \"41fe6d63-9316-4916-b042-87ba7e602167\") " pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf"
Oct 30 21:50:23.674309 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.674094    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7qxb\" (UniqueName: \"kubernetes.io/projected/49ff8cb6-54de-4e8c-9a35-634f9a1ef29b-kube-api-access-z7qxb\") pod \"dns-default-fdmdz\" (UID: \"49ff8cb6-54de-4e8c-9a35-634f9a1ef29b\") " pod="openshift-dns/dns-default-fdmdz"
Oct 30 21:50:23.674507 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.674467    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/a3459ebb-2fc2-4a82-b9c0-de5542d667e1-metrics-server-audit-profiles\") pod \"metrics-server-dc7c7d47d-ghdn7\" (UID: \"a3459ebb-2fc2-4a82-b9c0-de5542d667e1\") " pod="openshift-monitoring/metrics-server-dc7c7d47d-ghdn7"
Oct 30 21:50:23.674546 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.674538    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5f468f3c-bf3d-437c-8ec8-a5d56f448d99-multus-cni-dir\") pod \"multus-p7zpx\" (UID: \"5f468f3c-bf3d-437c-8ec8-a5d56f448d99\") " pod="openshift-multus/multus-p7zpx"
Oct 30 21:50:23.674594 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.674571    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/67b50e2a-4d01-4d5f-a5d1-7e80ee62642a-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-vcj4q\" (UID: \"67b50e2a-4d01-4d5f-a5d1-7e80ee62642a\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-vcj4q"
Oct 30 21:50:23.674627 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.674618    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/41fe6d63-9316-4916-b042-87ba7e602167-run-ovn\") pod \"ovnkube-node-ml9xf\" (UID: \"41fe6d63-9316-4916-b042-87ba7e602167\") " pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf"
Oct 30 21:50:23.674688 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.674667    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:23.674731 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.674719    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5138fe8c-9927-4d05-8fe5-5556072ed595-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5138fe8c-9927-4d05-8fe5-5556072ed595\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:50:23.674764 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.674751    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5f468f3c-bf3d-437c-8ec8-a5d56f448d99-host-run-k8s-cni-cncf-io\") pod \"multus-p7zpx\" (UID: \"5f468f3c-bf3d-437c-8ec8-a5d56f448d99\") " pod="openshift-multus/multus-p7zpx"
Oct 30 21:50:23.674792 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.674783    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fn582\" (UniqueName: \"kubernetes.io/projected/67b50e2a-4d01-4d5f-a5d1-7e80ee62642a-kube-api-access-fn582\") pod \"prometheus-operator-9cd89b74c-vcj4q\" (UID: \"67b50e2a-4d01-4d5f-a5d1-7e80ee62642a\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-vcj4q"
Oct 30 21:50:23.674824 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.674814    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/dec1cd54-fec6-4c12-9d37-a1b0b1a7ffc0-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-pvl2l\" (UID: \"dec1cd54-fec6-4c12-9d37-a1b0b1a7ffc0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pvl2l"
Oct 30 21:50:23.674853 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.674842    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/82f75dd5-40cd-4cb3-9111-6213d6b43679-host\") pod \"node-ca-gmbbw\" (UID: \"82f75dd5-40cd-4cb3-9111-6213d6b43679\") " pod="openshift-image-registry/node-ca-gmbbw"
Oct 30 21:50:23.674882 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.674864    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/41fe6d63-9316-4916-b042-87ba7e602167-var-lib-openvswitch\") pod \"ovnkube-node-ml9xf\" (UID: \"41fe6d63-9316-4916-b042-87ba7e602167\") " pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf"
Oct 30 21:50:23.674911 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.674892    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d947d3db-ab6e-4874-8817-f8c74be138a1-etc-sysctl-conf\") pod \"tuned-22pl2\" (UID: \"d947d3db-ab6e-4874-8817-f8c74be138a1\") " pod="openshift-cluster-node-tuning-operator/tuned-22pl2"
Oct 30 21:50:23.674948 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.674918    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d947d3db-ab6e-4874-8817-f8c74be138a1-host\") pod \"tuned-22pl2\" (UID: \"d947d3db-ab6e-4874-8817-f8c74be138a1\") " pod="openshift-cluster-node-tuning-operator/tuned-22pl2"
Oct 30 21:50:23.674996 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.674984    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/69e218ec-ab5e-4e09-8444-e53a710c53bf-oauth-serving-cert\") pod \"console-69874d4cdd-zhlp5\" (UID: \"69e218ec-ab5e-4e09-8444-e53a710c53bf\") " pod="openshift-console/console-69874d4cdd-zhlp5"
Oct 30 21:50:23.675036 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.675020    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/924394f7-5b3b-4767-a1ff-f1d310ece4ed-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lrc2c\" (UID: \"924394f7-5b3b-4767-a1ff-f1d310ece4ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lrc2c"
Oct 30 21:50:23.675082 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.675053    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b954bab2-9dd7-4b39-bda6-3fec87381aff-original-pull-secret\") pod \"global-pull-secret-syncer-cck7d\" (UID: \"b954bab2-9dd7-4b39-bda6-3fec87381aff\") " pod="kube-system/global-pull-secret-syncer-cck7d"
Oct 30 21:50:23.675111 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.675102    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxd2f\" (UniqueName: \"kubernetes.io/projected/32d63b18-0fcb-4e4a-a24c-ee044dcb5a90-kube-api-access-pxd2f\") pod \"node-resolver-gbpvx\" (UID: \"32d63b18-0fcb-4e4a-a24c-ee044dcb5a90\") " pod="openshift-dns/node-resolver-gbpvx"
Oct 30 21:50:23.675154 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.675133    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6v9rf\" (UniqueName: \"kubernetes.io/projected/5138fe8c-9927-4d05-8fe5-5556072ed595-kube-api-access-6v9rf\") pod \"alertmanager-main-0\" (UID: \"5138fe8c-9927-4d05-8fe5-5556072ed595\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:50:23.675249 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.675238    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0ad59239-edcc-46a4-a5a1-1debb63e3a5d-agent-certs\") pod \"konnectivity-agent-w72c9\" (UID: \"0ad59239-edcc-46a4-a5a1-1debb63e3a5d\") " pod="kube-system/konnectivity-agent-w72c9"
Oct 30 21:50:23.675279 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.675265    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/41fe6d63-9316-4916-b042-87ba7e602167-host-kubelet\") pod \"ovnkube-node-ml9xf\" (UID: \"41fe6d63-9316-4916-b042-87ba7e602167\") " pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf"
Oct 30 21:50:23.675322 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.675293    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/49ff8cb6-54de-4e8c-9a35-634f9a1ef29b-metrics-tls\") pod \"dns-default-fdmdz\" (UID: \"49ff8cb6-54de-4e8c-9a35-634f9a1ef29b\") " pod="openshift-dns/dns-default-fdmdz"
Oct 30 21:50:23.675352 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.675344    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5138fe8c-9927-4d05-8fe5-5556072ed595-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5138fe8c-9927-4d05-8fe5-5556072ed595\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:50:23.675428 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.675393    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:23.675461 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.675448    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:23.675490 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.675472    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d947d3db-ab6e-4874-8817-f8c74be138a1-etc-kubernetes\") pod \"tuned-22pl2\" (UID: \"d947d3db-ab6e-4874-8817-f8c74be138a1\") " pod="openshift-cluster-node-tuning-operator/tuned-22pl2"
Oct 30 21:50:23.675528 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.675508    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d947d3db-ab6e-4874-8817-f8c74be138a1-sys\") pod \"tuned-22pl2\" (UID: \"d947d3db-ab6e-4874-8817-f8c74be138a1\") " pod="openshift-cluster-node-tuning-operator/tuned-22pl2"
Oct 30 21:50:23.675558 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.675552    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/516573f9-4ada-4868-b885-f61b9640e6bf-cert\") pod \"ingress-canary-hr227\" (UID: \"516573f9-4ada-4868-b885-f61b9640e6bf\") " pod="openshift-ingress-canary/ingress-canary-hr227"
Oct 30 21:50:23.675592 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.675581    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b954bab2-9dd7-4b39-bda6-3fec87381aff-kubelet-config\") pod \"global-pull-secret-syncer-cck7d\" (UID: \"b954bab2-9dd7-4b39-bda6-3fec87381aff\") " pod="kube-system/global-pull-secret-syncer-cck7d"
Oct 30 21:50:23.675622 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.675615    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/41fe6d63-9316-4916-b042-87ba7e602167-host-cni-bin\") pod \"ovnkube-node-ml9xf\" (UID: \"41fe6d63-9316-4916-b042-87ba7e602167\") " pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf"
Oct 30 21:50:23.675650 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.675643    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/41fe6d63-9316-4916-b042-87ba7e602167-ovnkube-config\") pod \"ovnkube-node-ml9xf\" (UID: \"41fe6d63-9316-4916-b042-87ba7e602167\") " pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf"
Oct 30 21:50:23.676010 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.675993    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:23.676054 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.676033    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d947d3db-ab6e-4874-8817-f8c74be138a1-etc-tuned\") pod \"tuned-22pl2\" (UID: \"d947d3db-ab6e-4874-8817-f8c74be138a1\") " pod="openshift-cluster-node-tuning-operator/tuned-22pl2"
Oct 30 21:50:23.676088 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.676078    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5f468f3c-bf3d-437c-8ec8-a5d56f448d99-hostroot\") pod \"multus-p7zpx\" (UID: \"5f468f3c-bf3d-437c-8ec8-a5d56f448d99\") " pod="openshift-multus/multus-p7zpx"
Oct 30 21:50:23.676124 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.676112    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dec1cd54-fec6-4c12-9d37-a1b0b1a7ffc0-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-pvl2l\" (UID: \"dec1cd54-fec6-4c12-9d37-a1b0b1a7ffc0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pvl2l"
Oct 30 21:50:23.676160 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.676142    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/41fe6d63-9316-4916-b042-87ba7e602167-log-socket\") pod \"ovnkube-node-ml9xf\" (UID: \"41fe6d63-9316-4916-b042-87ba7e602167\") " pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf"
Oct 30 21:50:23.676195 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.676174    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7a6305de-8fd1-4d3e-9628-8c4cda07f379-system-cni-dir\") pod \"multus-additional-cni-plugins-vrtnk\" (UID: \"7a6305de-8fd1-4d3e-9628-8c4cda07f379\") " pod="openshift-multus/multus-additional-cni-plugins-vrtnk"
Oct 30 21:50:23.676225 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.676216    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5138fe8c-9927-4d05-8fe5-5556072ed595-config-volume\") pod \"alertmanager-main-0\" (UID: \"5138fe8c-9927-4d05-8fe5-5556072ed595\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:50:23.676257 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.676248    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/a3459ebb-2fc2-4a82-b9c0-de5542d667e1-audit-log\") pod \"metrics-server-dc7c7d47d-ghdn7\" (UID: \"a3459ebb-2fc2-4a82-b9c0-de5542d667e1\") " pod="openshift-monitoring/metrics-server-dc7c7d47d-ghdn7"
Oct 30 21:50:23.676285 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.676274    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fdf54bb9-f735-4810-82ee-d803099a1bcf-console-oauth-config\") pod \"console-7cbdbcf4b5-rmbsz\" (UID: \"fdf54bb9-f735-4810-82ee-d803099a1bcf\") " pod="openshift-console/console-7cbdbcf4b5-rmbsz"
Oct 30 21:50:23.676313 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.676304    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/924394f7-5b3b-4767-a1ff-f1d310ece4ed-socket-dir\") pod \"aws-ebs-csi-driver-node-lrc2c\" (UID: \"924394f7-5b3b-4767-a1ff-f1d310ece4ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lrc2c"
Oct 30 21:50:23.676341 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.676333    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f938a93b-29dd-4e7d-b8e9-444df68acf03-root\") pod \"node-exporter-7j5j6\" (UID: \"f938a93b-29dd-4e7d-b8e9-444df68acf03\") " pod="openshift-monitoring/node-exporter-7j5j6"
Oct 30 21:50:23.676372 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.676365    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jt5t\" (UniqueName: \"kubernetes.io/projected/f938a93b-29dd-4e7d-b8e9-444df68acf03-kube-api-access-6jt5t\") pod \"node-exporter-7j5j6\" (UID: \"f938a93b-29dd-4e7d-b8e9-444df68acf03\") " pod="openshift-monitoring/node-exporter-7j5j6"
Oct 30 21:50:23.676450 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.676435    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8w8vm\" (UniqueName: \"kubernetes.io/projected/f22e4514-ac6d-481d-8afe-6e2e4cdc3aa1-kube-api-access-8w8vm\") pod \"iptables-alerter-t24lz\" (UID: \"f22e4514-ac6d-481d-8afe-6e2e4cdc3aa1\") " pod="openshift-network-operator/iptables-alerter-t24lz"
Oct 30 21:50:23.676481 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.676470    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dec1cd54-fec6-4c12-9d37-a1b0b1a7ffc0-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-pvl2l\" (UID: \"dec1cd54-fec6-4c12-9d37-a1b0b1a7ffc0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pvl2l"
Oct 30 21:50:23.676511 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.676502    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5138fe8c-9927-4d05-8fe5-5556072ed595-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5138fe8c-9927-4d05-8fe5-5556072ed595\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:50:23.676545 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.676533    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-web-config\") pod \"prometheus-k8s-0\" (UID: \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:23.676579 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.676568    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/87c44883-54d0-4a05-b94b-d3c2835a03e1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kn6xx\" (UID: \"87c44883-54d0-4a05-b94b-d3c2835a03e1\") " pod="openshift-insights/insights-runtime-extractor-kn6xx"
Oct 30 21:50:23.676608 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.676600    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5f468f3c-bf3d-437c-8ec8-a5d56f448d99-host-var-lib-kubelet\") pod \"multus-p7zpx\" (UID: \"5f468f3c-bf3d-437c-8ec8-a5d56f448d99\") " pod="openshift-multus/multus-p7zpx"
Oct 30 21:50:23.676636 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.676628    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f938a93b-29dd-4e7d-b8e9-444df68acf03-metrics-client-ca\") pod \"node-exporter-7j5j6\" (UID: \"f938a93b-29dd-4e7d-b8e9-444df68acf03\") " pod="openshift-monitoring/node-exporter-7j5j6"
Oct 30 21:50:23.676667 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.676659    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/41fe6d63-9316-4916-b042-87ba7e602167-etc-openvswitch\") pod \"ovnkube-node-ml9xf\" (UID: \"41fe6d63-9316-4916-b042-87ba7e602167\") " pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf"
Oct 30 21:50:23.676703 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.676690    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/41fe6d63-9316-4916-b042-87ba7e602167-ovn-node-metrics-cert\") pod \"ovnkube-node-ml9xf\" (UID: \"41fe6d63-9316-4916-b042-87ba7e602167\") " pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf"
Oct 30 21:50:23.676735 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.676723    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/87c44883-54d0-4a05-b94b-d3c2835a03e1-crio-socket\") pod \"insights-runtime-extractor-kn6xx\" (UID: \"87c44883-54d0-4a05-b94b-d3c2835a03e1\") " pod="openshift-insights/insights-runtime-extractor-kn6xx"
Oct 30 21:50:23.676764 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.676747    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/924394f7-5b3b-4767-a1ff-f1d310ece4ed-registration-dir\") pod \"aws-ebs-csi-driver-node-lrc2c\" (UID: \"924394f7-5b3b-4767-a1ff-f1d310ece4ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lrc2c"
Oct 30 21:50:23.676792 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.676780    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnt2l\" (UniqueName: \"kubernetes.io/projected/924394f7-5b3b-4767-a1ff-f1d310ece4ed-kube-api-access-qnt2l\") pod \"aws-ebs-csi-driver-node-lrc2c\" (UID: \"924394f7-5b3b-4767-a1ff-f1d310ece4ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lrc2c"
Oct 30 21:50:23.676820 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.676812    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5f468f3c-bf3d-437c-8ec8-a5d56f448d99-host-run-netns\") pod \"multus-p7zpx\" (UID: \"5f468f3c-bf3d-437c-8ec8-a5d56f448d99\") " pod="openshift-multus/multus-p7zpx"
Oct 30 21:50:23.676859 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.676844    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvbdq\" (UniqueName: \"kubernetes.io/projected/dec1cd54-fec6-4c12-9d37-a1b0b1a7ffc0-kube-api-access-wvbdq\") pod \"kube-state-metrics-5dc8d7d89b-pvl2l\" (UID: \"dec1cd54-fec6-4c12-9d37-a1b0b1a7ffc0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pvl2l"
Oct 30 21:50:23.676891 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.676881    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/41fe6d63-9316-4916-b042-87ba7e602167-host-run-ovn-kubernetes\") pod \"ovnkube-node-ml9xf\" (UID: \"41fe6d63-9316-4916-b042-87ba7e602167\") " pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf"
Oct 30 21:50:23.676923 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.676909    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/32d63b18-0fcb-4e4a-a24c-ee044dcb5a90-hosts-file\") pod \"node-resolver-gbpvx\" (UID: \"32d63b18-0fcb-4e4a-a24c-ee044dcb5a90\") " pod="openshift-dns/node-resolver-gbpvx"
Oct 30 21:50:23.676951 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.676941    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:23.676986 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.676973    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d947d3db-ab6e-4874-8817-f8c74be138a1-lib-modules\") pod \"tuned-22pl2\" (UID: \"d947d3db-ab6e-4874-8817-f8c74be138a1\") " pod="openshift-cluster-node-tuning-operator/tuned-22pl2"
Oct 30 21:50:23.677019 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.677010    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d947d3db-ab6e-4874-8817-f8c74be138a1-var-lib-kubelet\") pod \"tuned-22pl2\" (UID: \"d947d3db-ab6e-4874-8817-f8c74be138a1\") " pod="openshift-cluster-node-tuning-operator/tuned-22pl2"
Oct 30 21:50:23.677056 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.677043    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gv6wf\" (UniqueName: \"kubernetes.io/projected/69e218ec-ab5e-4e09-8444-e53a710c53bf-kube-api-access-gv6wf\") pod \"console-69874d4cdd-zhlp5\" (UID: \"69e218ec-ab5e-4e09-8444-e53a710c53bf\") " pod="openshift-console/console-69874d4cdd-zhlp5"
Oct 30 21:50:23.677085 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.677071    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/41fe6d63-9316-4916-b042-87ba7e602167-host-slash\") pod \"ovnkube-node-ml9xf\" (UID: \"41fe6d63-9316-4916-b042-87ba7e602167\") " pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf"
Oct 30 21:50:23.677113 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.677103    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/41fe6d63-9316-4916-b042-87ba7e602167-host-run-netns\") pod \"ovnkube-node-ml9xf\" (UID: \"41fe6d63-9316-4916-b042-87ba7e602167\") " pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf"
Oct 30 21:50:23.677148 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.677134    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/41fe6d63-9316-4916-b042-87ba7e602167-run-systemd\") pod \"ovnkube-node-ml9xf\" (UID: \"41fe6d63-9316-4916-b042-87ba7e602167\") " pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf"
Oct 30 21:50:23.677181 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.677172    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7a6305de-8fd1-4d3e-9628-8c4cda07f379-cni-binary-copy\") pod \"multus-additional-cni-plugins-vrtnk\" (UID: \"7a6305de-8fd1-4d3e-9628-8c4cda07f379\") " pod="openshift-multus/multus-additional-cni-plugins-vrtnk"
Oct 30 21:50:23.677210 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.677200    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/69e218ec-ab5e-4e09-8444-e53a710c53bf-console-config\") pod \"console-69874d4cdd-zhlp5\" (UID: \"69e218ec-ab5e-4e09-8444-e53a710c53bf\") " pod="openshift-console/console-69874d4cdd-zhlp5"
Oct 30 21:50:23.677245 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.677233    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fdf54bb9-f735-4810-82ee-d803099a1bcf-oauth-serving-cert\") pod \"console-7cbdbcf4b5-rmbsz\" (UID: \"fdf54bb9-f735-4810-82ee-d803099a1bcf\") " pod="openshift-console/console-7cbdbcf4b5-rmbsz"
Oct 30 21:50:23.677280 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.677268    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/a3459ebb-2fc2-4a82-b9c0-de5542d667e1-secret-metrics-server-client-certs\") pod \"metrics-server-dc7c7d47d-ghdn7\" (UID: \"a3459ebb-2fc2-4a82-b9c0-de5542d667e1\") " pod="openshift-monitoring/metrics-server-dc7c7d47d-ghdn7"
Oct 30 21:50:23.677317 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.677304    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hdbq\" (UniqueName: \"kubernetes.io/projected/a3459ebb-2fc2-4a82-b9c0-de5542d667e1-kube-api-access-6hdbq\") pod \"metrics-server-dc7c7d47d-ghdn7\" (UID: \"a3459ebb-2fc2-4a82-b9c0-de5542d667e1\") " pod="openshift-monitoring/metrics-server-dc7c7d47d-ghdn7"
Oct 30 21:50:23.677382 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.677339    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/924394f7-5b3b-4767-a1ff-f1d310ece4ed-etc-selinux\") pod \"aws-ebs-csi-driver-node-lrc2c\" (UID: \"924394f7-5b3b-4767-a1ff-f1d310ece4ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lrc2c"
Oct 30 21:50:23.677382 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.677367    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czfxj\" (UniqueName: \"kubernetes.io/projected/a6980f40-89cf-4f3d-8cd5-91969ae5a276-kube-api-access-czfxj\") pod \"network-metrics-daemon-xbh9m\" (UID: \"a6980f40-89cf-4f3d-8cd5-91969ae5a276\") " pod="openshift-multus/network-metrics-daemon-xbh9m"
Oct 30 21:50:23.677450 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.677415    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5f468f3c-bf3d-437c-8ec8-a5d56f448d99-multus-socket-dir-parent\") pod \"multus-p7zpx\" (UID: \"5f468f3c-bf3d-437c-8ec8-a5d56f448d99\") " pod="openshift-multus/multus-p7zpx"
Oct 30 21:50:23.677480 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.677450    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:23.677536 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.677484    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:23.677536 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.677519    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66wt8\" (UniqueName: \"kubernetes.io/projected/516573f9-4ada-4868-b885-f61b9640e6bf-kube-api-access-66wt8\") pod \"ingress-canary-hr227\" (UID: \"516573f9-4ada-4868-b885-f61b9640e6bf\") " pod="openshift-ingress-canary/ingress-canary-hr227"
Oct 30 21:50:23.677774 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.677755    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f938a93b-29dd-4e7d-b8e9-444df68acf03-metrics-client-ca\") pod \"node-exporter-7j5j6\" (UID: \"f938a93b-29dd-4e7d-b8e9-444df68acf03\") " pod="openshift-monitoring/node-exporter-7j5j6"
Oct 30 21:50:23.677838 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.677783    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5138fe8c-9927-4d05-8fe5-5556072ed595-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5138fe8c-9927-4d05-8fe5-5556072ed595\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:50:23.677888 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.677851    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:23.677888 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.677860    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/41fe6d63-9316-4916-b042-87ba7e602167-ovn-node-metrics-cert\") pod \"ovnkube-node-ml9xf\" (UID: \"41fe6d63-9316-4916-b042-87ba7e602167\") " pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf"
Oct 30 21:50:23.678021 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.678001    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5138fe8c-9927-4d05-8fe5-5556072ed595-config-volume\") pod \"alertmanager-main-0\" (UID: \"5138fe8c-9927-4d05-8fe5-5556072ed595\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:50:23.678098 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.678082    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/a3459ebb-2fc2-4a82-b9c0-de5542d667e1-audit-log\") pod \"metrics-server-dc7c7d47d-ghdn7\" (UID: \"a3459ebb-2fc2-4a82-b9c0-de5542d667e1\") " pod="openshift-monitoring/metrics-server-dc7c7d47d-ghdn7"
Oct 30 21:50:23.678197 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.678181    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5138fe8c-9927-4d05-8fe5-5556072ed595-web-config\") pod \"alertmanager-main-0\" (UID: \"5138fe8c-9927-4d05-8fe5-5556072ed595\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:50:23.678251 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.678224    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5138fe8c-9927-4d05-8fe5-5556072ed595-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5138fe8c-9927-4d05-8fe5-5556072ed595\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:50:23.678304 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.678262    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/69e218ec-ab5e-4e09-8444-e53a710c53bf-console-oauth-config\") pod \"console-69874d4cdd-zhlp5\" (UID: \"69e218ec-ab5e-4e09-8444-e53a710c53bf\") " pod="openshift-console/console-69874d4cdd-zhlp5"
Oct 30 21:50:23.678304 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.678296    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/41fe6d63-9316-4916-b042-87ba7e602167-node-log\") pod \"ovnkube-node-ml9xf\" (UID: \"41fe6d63-9316-4916-b042-87ba7e602167\") " pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf"
Oct 30 21:50:23.678426 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.678325    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49ff8cb6-54de-4e8c-9a35-634f9a1ef29b-tmp-dir\") pod \"dns-default-fdmdz\" (UID: \"49ff8cb6-54de-4e8c-9a35-634f9a1ef29b\") " pod="openshift-dns/dns-default-fdmdz"
Oct 30 21:50:23.678426 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.678358    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-config\") pod \"prometheus-k8s-0\" (UID: \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:23.678426 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.678390    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:23.678581 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.678440    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d947d3db-ab6e-4874-8817-f8c74be138a1-tmp\") pod \"tuned-22pl2\" (UID: \"d947d3db-ab6e-4874-8817-f8c74be138a1\") " pod="openshift-cluster-node-tuning-operator/tuned-22pl2"
Oct 30 21:50:23.678581 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.678479    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6x28\" (UniqueName: \"kubernetes.io/projected/55619a2e-f6e9-4719-b362-30b69d553967-kube-api-access-j6x28\") pod \"network-check-target-w9tx8\" (UID: \"55619a2e-f6e9-4719-b362-30b69d553967\") " pod="openshift-network-diagnostics/network-check-target-w9tx8"
Oct 30 21:50:23.678581 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.678511    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/67b50e2a-4d01-4d5f-a5d1-7e80ee62642a-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-vcj4q\" (UID: \"67b50e2a-4d01-4d5f-a5d1-7e80ee62642a\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-vcj4q"
Oct 30 21:50:23.678581 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.678524    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fdf54bb9-f735-4810-82ee-d803099a1bcf-console-oauth-config\") pod \"console-7cbdbcf4b5-rmbsz\" (UID: \"fdf54bb9-f735-4810-82ee-d803099a1bcf\") " pod="openshift-console/console-7cbdbcf4b5-rmbsz"
Oct 30 21:50:23.678581 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.678539    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:23.678813 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.678721    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:23.678813 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.678783    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fdf54bb9-f735-4810-82ee-d803099a1bcf-oauth-serving-cert\") pod \"console-7cbdbcf4b5-rmbsz\" (UID: \"fdf54bb9-f735-4810-82ee-d803099a1bcf\") " pod="openshift-console/console-7cbdbcf4b5-rmbsz"
Oct 30 21:50:23.678907 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.678807    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/516573f9-4ada-4868-b885-f61b9640e6bf-cert\") pod \"ingress-canary-hr227\" (UID: \"516573f9-4ada-4868-b885-f61b9640e6bf\") " pod="openshift-ingress-canary/ingress-canary-hr227"
Oct 30 21:50:23.678907 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.678840    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5138fe8c-9927-4d05-8fe5-5556072ed595-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5138fe8c-9927-4d05-8fe5-5556072ed595\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:50:23.679004 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.678937    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:23.679059 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.679034    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dec1cd54-fec6-4c12-9d37-a1b0b1a7ffc0-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-pvl2l\" (UID: \"dec1cd54-fec6-4c12-9d37-a1b0b1a7ffc0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pvl2l"
Oct 30 21:50:23.679059 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.679039    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/a3459ebb-2fc2-4a82-b9c0-de5542d667e1-secret-metrics-server-client-certs\") pod \"metrics-server-dc7c7d47d-ghdn7\" (UID: \"a3459ebb-2fc2-4a82-b9c0-de5542d667e1\") " pod="openshift-monitoring/metrics-server-dc7c7d47d-ghdn7"
Oct 30 21:50:23.679153 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.678545    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7a6305de-8fd1-4d3e-9628-8c4cda07f379-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vrtnk\" (UID: \"7a6305de-8fd1-4d3e-9628-8c4cda07f379\") " pod="openshift-multus/multus-additional-cni-plugins-vrtnk"
Oct 30 21:50:23.679153 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.679099    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d947d3db-ab6e-4874-8817-f8c74be138a1-etc-tuned\") pod \"tuned-22pl2\" (UID: \"d947d3db-ab6e-4874-8817-f8c74be138a1\") " pod="openshift-cluster-node-tuning-operator/tuned-22pl2"
Oct 30 21:50:23.679153 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.679108    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f938a93b-29dd-4e7d-b8e9-444df68acf03-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7j5j6\" (UID: \"f938a93b-29dd-4e7d-b8e9-444df68acf03\") " pod="openshift-monitoring/node-exporter-7j5j6"
Oct 30 21:50:23.679153 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.679137    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/41fe6d63-9316-4916-b042-87ba7e602167-ovnkube-config\") pod \"ovnkube-node-ml9xf\" (UID: \"41fe6d63-9316-4916-b042-87ba7e602167\") " pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf"
Oct 30 21:50:23.679347 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.679152    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/41fe6d63-9316-4916-b042-87ba7e602167-ovnkube-script-lib\") pod \"ovnkube-node-ml9xf\" (UID: \"41fe6d63-9316-4916-b042-87ba7e602167\") " pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf"
Oct 30 21:50:23.679347 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.679193    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7a6305de-8fd1-4d3e-9628-8c4cda07f379-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vrtnk\" (UID: \"7a6305de-8fd1-4d3e-9628-8c4cda07f379\") " pod="openshift-multus/multus-additional-cni-plugins-vrtnk"
Oct 30 21:50:23.679347 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.679227    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/87c44883-54d0-4a05-b94b-d3c2835a03e1-data-volume\") pod \"insights-runtime-extractor-kn6xx\" (UID: \"87c44883-54d0-4a05-b94b-d3c2835a03e1\") " pod="openshift-insights/insights-runtime-extractor-kn6xx"
Oct 30 21:50:23.679347 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.679265    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/87c44883-54d0-4a05-b94b-d3c2835a03e1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kn6xx\" (UID: \"87c44883-54d0-4a05-b94b-d3c2835a03e1\") " pod="openshift-insights/insights-runtime-extractor-kn6xx"
Oct 30 21:50:23.679347 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.679303    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5f468f3c-bf3d-437c-8ec8-a5d56f448d99-etc-kubernetes\") pod \"multus-p7zpx\" (UID: \"5f468f3c-bf3d-437c-8ec8-a5d56f448d99\") " pod="openshift-multus/multus-p7zpx"
Oct 30 21:50:23.679347 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.679330    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:23.679347 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.679341    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5138fe8c-9927-4d05-8fe5-5556072ed595-config-out\") pod \"alertmanager-main-0\" (UID: \"5138fe8c-9927-4d05-8fe5-5556072ed595\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:50:23.679681 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.679612    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/67b50e2a-4d01-4d5f-a5d1-7e80ee62642a-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-vcj4q\" (UID: \"67b50e2a-4d01-4d5f-a5d1-7e80ee62642a\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-vcj4q"
Oct 30 21:50:23.679681 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.679334    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-web-config\") pod \"prometheus-k8s-0\" (UID: \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:23.679829 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.679808    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/87c44883-54d0-4a05-b94b-d3c2835a03e1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kn6xx\" (UID: \"87c44883-54d0-4a05-b94b-d3c2835a03e1\") " pod="openshift-insights/insights-runtime-extractor-kn6xx"
Oct 30 21:50:23.679987 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.679906    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49ff8cb6-54de-4e8c-9a35-634f9a1ef29b-tmp-dir\") pod \"dns-default-fdmdz\" (UID: \"49ff8cb6-54de-4e8c-9a35-634f9a1ef29b\") " pod="openshift-dns/dns-default-fdmdz"
Oct 30 21:50:23.680049 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.679985    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:23.680049 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.679966    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f938a93b-29dd-4e7d-b8e9-444df68acf03-node-exporter-tls\") pod \"node-exporter-7j5j6\" (UID: \"f938a93b-29dd-4e7d-b8e9-444df68acf03\") " pod="openshift-monitoring/node-exporter-7j5j6"
Oct 30 21:50:23.680049 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.679973    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5138fe8c-9927-4d05-8fe5-5556072ed595-config-out\") pod \"alertmanager-main-0\" (UID: \"5138fe8c-9927-4d05-8fe5-5556072ed595\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:50:23.680243 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.680226    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/69e218ec-ab5e-4e09-8444-e53a710c53bf-console-serving-cert\") pod \"console-69874d4cdd-zhlp5\" (UID: \"69e218ec-ab5e-4e09-8444-e53a710c53bf\") " pod="openshift-console/console-69874d4cdd-zhlp5"
Oct 30 21:50:23.680303 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.680236    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/a3459ebb-2fc2-4a82-b9c0-de5542d667e1-secret-metrics-server-tls\") pod \"metrics-server-dc7c7d47d-ghdn7\" (UID: \"a3459ebb-2fc2-4a82-b9c0-de5542d667e1\") " pod="openshift-monitoring/metrics-server-dc7c7d47d-ghdn7"
Oct 30 21:50:23.680303 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.680258    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/69e218ec-ab5e-4e09-8444-e53a710c53bf-console-config\") pod \"console-69874d4cdd-zhlp5\" (UID: \"69e218ec-ab5e-4e09-8444-e53a710c53bf\") " pod="openshift-console/console-69874d4cdd-zhlp5"
Oct 30 21:50:23.680422 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.680304    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkkmr\" (UniqueName: \"kubernetes.io/projected/d947d3db-ab6e-4874-8817-f8c74be138a1-kube-api-access-mkkmr\") pod \"tuned-22pl2\" (UID: \"d947d3db-ab6e-4874-8817-f8c74be138a1\") " pod="openshift-cluster-node-tuning-operator/tuned-22pl2"
Oct 30 21:50:23.680422 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.680316    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f938a93b-29dd-4e7d-b8e9-444df68acf03-node-exporter-textfile\") pod \"node-exporter-7j5j6\" (UID: \"f938a93b-29dd-4e7d-b8e9-444df68acf03\") " pod="openshift-monitoring/node-exporter-7j5j6"
Oct 30 21:50:23.680422 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.680342    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/69e218ec-ab5e-4e09-8444-e53a710c53bf-service-ca\") pod \"console-69874d4cdd-zhlp5\" (UID: \"69e218ec-ab5e-4e09-8444-e53a710c53bf\") " pod="openshift-console/console-69874d4cdd-zhlp5"
Oct 30 21:50:23.680422 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.680385    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5f468f3c-bf3d-437c-8ec8-a5d56f448d99-multus-daemon-config\") pod \"multus-p7zpx\" (UID: \"5f468f3c-bf3d-437c-8ec8-a5d56f448d99\") " pod="openshift-multus/multus-p7zpx"
Oct 30 21:50:23.680631 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.680446    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cr8x2\" (UniqueName: \"kubernetes.io/projected/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-kube-api-access-cr8x2\") pod \"prometheus-k8s-0\" (UID: \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:23.680631 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.680468    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dec1cd54-fec6-4c12-9d37-a1b0b1a7ffc0-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-pvl2l\" (UID: \"dec1cd54-fec6-4c12-9d37-a1b0b1a7ffc0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pvl2l"
Oct 30 21:50:23.680631 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.680487    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/67b50e2a-4d01-4d5f-a5d1-7e80ee62642a-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-vcj4q\" (UID: \"67b50e2a-4d01-4d5f-a5d1-7e80ee62642a\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-vcj4q"
Oct 30 21:50:23.680631 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.680575    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/69e218ec-ab5e-4e09-8444-e53a710c53bf-tmp\") pod \"console-69874d4cdd-zhlp5\" (UID: \"69e218ec-ab5e-4e09-8444-e53a710c53bf\") " pod="openshift-console/console-69874d4cdd-zhlp5"
Oct 30 21:50:23.680631 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.680580    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/41fe6d63-9316-4916-b042-87ba7e602167-ovnkube-script-lib\") pod \"ovnkube-node-ml9xf\" (UID: \"41fe6d63-9316-4916-b042-87ba7e602167\") " pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf"
Oct 30 21:50:23.680860 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.680674    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7a6305de-8fd1-4d3e-9628-8c4cda07f379-cni-binary-copy\") pod \"multus-additional-cni-plugins-vrtnk\" (UID: \"7a6305de-8fd1-4d3e-9628-8c4cda07f379\") " pod="openshift-multus/multus-additional-cni-plugins-vrtnk"
Oct 30 21:50:23.680860 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.680678    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/69e218ec-ab5e-4e09-8444-e53a710c53bf-console-oauth-config\") pod \"console-69874d4cdd-zhlp5\" (UID: \"69e218ec-ab5e-4e09-8444-e53a710c53bf\") " pod="openshift-console/console-69874d4cdd-zhlp5"
Oct 30 21:50:23.680860 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.680491    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b954bab2-9dd7-4b39-bda6-3fec87381aff-dbus\") pod \"global-pull-secret-syncer-cck7d\" (UID: \"b954bab2-9dd7-4b39-bda6-3fec87381aff\") " pod="kube-system/global-pull-secret-syncer-cck7d"
Oct 30 21:50:23.680860 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.680703    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fdf54bb9-f735-4810-82ee-d803099a1bcf-service-ca\") pod \"console-7cbdbcf4b5-rmbsz\" (UID: \"fdf54bb9-f735-4810-82ee-d803099a1bcf\") " pod="openshift-console/console-7cbdbcf4b5-rmbsz"
Oct 30 21:50:23.680860 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.680741    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/41fe6d63-9316-4916-b042-87ba7e602167-systemd-units\") pod \"ovnkube-node-ml9xf\" (UID: \"41fe6d63-9316-4916-b042-87ba7e602167\") " pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf"
Oct 30 21:50:23.680860 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.680771    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f938a93b-29dd-4e7d-b8e9-444df68acf03-node-exporter-accelerators-collector-config\") pod \"node-exporter-7j5j6\" (UID: \"f938a93b-29dd-4e7d-b8e9-444df68acf03\") " pod="openshift-monitoring/node-exporter-7j5j6"
Oct 30 21:50:23.680860 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.680779    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/41fe6d63-9316-4916-b042-87ba7e602167-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ml9xf\" (UID: \"41fe6d63-9316-4916-b042-87ba7e602167\") " pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf"
Oct 30 21:50:23.680860 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.680809    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/69e218ec-ab5e-4e09-8444-e53a710c53bf-oauth-serving-cert\") pod \"console-69874d4cdd-zhlp5\" (UID: \"69e218ec-ab5e-4e09-8444-e53a710c53bf\") " pod="openshift-console/console-69874d4cdd-zhlp5"
Oct 30 21:50:23.680860 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.680823    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwvcm\" (UniqueName: \"kubernetes.io/projected/7a6305de-8fd1-4d3e-9628-8c4cda07f379-kube-api-access-lwvcm\") pod \"multus-additional-cni-plugins-vrtnk\" (UID: \"7a6305de-8fd1-4d3e-9628-8c4cda07f379\") " pod="openshift-multus/multus-additional-cni-plugins-vrtnk"
Oct 30 21:50:23.680860 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.680860    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/49ff8cb6-54de-4e8c-9a35-634f9a1ef29b-config-volume\") pod \"dns-default-fdmdz\" (UID: \"49ff8cb6-54de-4e8c-9a35-634f9a1ef29b\") " pod="openshift-dns/dns-default-fdmdz"
Oct 30 21:50:23.681353 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.680870    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5138fe8c-9927-4d05-8fe5-5556072ed595-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5138fe8c-9927-4d05-8fe5-5556072ed595\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:50:23.681353 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.680919    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5138fe8c-9927-4d05-8fe5-5556072ed595-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5138fe8c-9927-4d05-8fe5-5556072ed595\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:50:23.681353 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.680968    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3459ebb-2fc2-4a82-b9c0-de5542d667e1-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-dc7c7d47d-ghdn7\" (UID: \"a3459ebb-2fc2-4a82-b9c0-de5542d667e1\") " pod="openshift-monitoring/metrics-server-dc7c7d47d-ghdn7"
Oct 30 21:50:23.681353 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.680970    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5138fe8c-9927-4d05-8fe5-5556072ed595-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5138fe8c-9927-4d05-8fe5-5556072ed595\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:50:23.681353 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.681013    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fdf54bb9-f735-4810-82ee-d803099a1bcf-trusted-ca-bundle\") pod \"console-7cbdbcf4b5-rmbsz\" (UID: \"fdf54bb9-f735-4810-82ee-d803099a1bcf\") " pod="openshift-console/console-7cbdbcf4b5-rmbsz"
Oct 30 21:50:23.681353 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.681035    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/67b50e2a-4d01-4d5f-a5d1-7e80ee62642a-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-vcj4q\" (UID: \"67b50e2a-4d01-4d5f-a5d1-7e80ee62642a\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-vcj4q"
Oct 30 21:50:23.681353 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.681051    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5138fe8c-9927-4d05-8fe5-5556072ed595-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5138fe8c-9927-4d05-8fe5-5556072ed595\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:50:23.681353 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.681086    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/49ff8cb6-54de-4e8c-9a35-634f9a1ef29b-config-volume\") pod \"dns-default-fdmdz\" (UID: \"49ff8cb6-54de-4e8c-9a35-634f9a1ef29b\") " pod="openshift-dns/dns-default-fdmdz"
Oct 30 21:50:23.681353 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.681092    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-config-out\") pod \"prometheus-k8s-0\" (UID: \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:23.681353 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.681168    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3459ebb-2fc2-4a82-b9c0-de5542d667e1-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-dc7c7d47d-ghdn7\" (UID: \"a3459ebb-2fc2-4a82-b9c0-de5542d667e1\") " pod="openshift-monitoring/metrics-server-dc7c7d47d-ghdn7"
Oct 30 21:50:23.681353 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.681186    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-config-out\") pod \"prometheus-k8s-0\" (UID: \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:23.681353 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.681234    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/dec1cd54-fec6-4c12-9d37-a1b0b1a7ffc0-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-pvl2l\" (UID: \"dec1cd54-fec6-4c12-9d37-a1b0b1a7ffc0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pvl2l"
Oct 30 21:50:23.681353 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.681280    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7a6305de-8fd1-4d3e-9628-8c4cda07f379-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vrtnk\" (UID: \"7a6305de-8fd1-4d3e-9628-8c4cda07f379\") " pod="openshift-multus/multus-additional-cni-plugins-vrtnk"
Oct 30 21:50:23.682020 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.681523    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-config\") pod \"prometheus-k8s-0\" (UID: \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:23.682020 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.681697    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f938a93b-29dd-4e7d-b8e9-444df68acf03-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7j5j6\" (UID: \"f938a93b-29dd-4e7d-b8e9-444df68acf03\") " pod="openshift-monitoring/node-exporter-7j5j6"
Oct 30 21:50:23.682020 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.681715    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:23.682020 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.681729    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/a3459ebb-2fc2-4a82-b9c0-de5542d667e1-metrics-server-audit-profiles\") pod \"metrics-server-dc7c7d47d-ghdn7\" (UID: \"a3459ebb-2fc2-4a82-b9c0-de5542d667e1\") " pod="openshift-monitoring/metrics-server-dc7c7d47d-ghdn7"
Oct 30 21:50:23.682020 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.681777    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5138fe8c-9927-4d05-8fe5-5556072ed595-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5138fe8c-9927-4d05-8fe5-5556072ed595\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:50:23.682020 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.681786    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d947d3db-ab6e-4874-8817-f8c74be138a1-etc-sysconfig\") pod \"tuned-22pl2\" (UID: \"d947d3db-ab6e-4874-8817-f8c74be138a1\") " pod="openshift-cluster-node-tuning-operator/tuned-22pl2"
Oct 30 21:50:23.682020 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.681828    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/87c44883-54d0-4a05-b94b-d3c2835a03e1-data-volume\") pod \"insights-runtime-extractor-kn6xx\" (UID: \"87c44883-54d0-4a05-b94b-d3c2835a03e1\") " pod="openshift-insights/insights-runtime-extractor-kn6xx"
Oct 30 21:50:23.682020 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.681901    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32d63b18-0fcb-4e4a-a24c-ee044dcb5a90-tmp-dir\") pod \"node-resolver-gbpvx\" (UID: \"32d63b18-0fcb-4e4a-a24c-ee044dcb5a90\") " pod="openshift-dns/node-resolver-gbpvx"
Oct 30 21:50:23.682020 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.681978    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5138fe8c-9927-4d05-8fe5-5556072ed595-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5138fe8c-9927-4d05-8fe5-5556072ed595\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:50:23.682524 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.682045    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdznq\" (UniqueName: \"kubernetes.io/projected/87c44883-54d0-4a05-b94b-d3c2835a03e1-kube-api-access-cdznq\") pod \"insights-runtime-extractor-kn6xx\" (UID: \"87c44883-54d0-4a05-b94b-d3c2835a03e1\") " pod="openshift-insights/insights-runtime-extractor-kn6xx"
Oct 30 21:50:23.682524 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.682051    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5138fe8c-9927-4d05-8fe5-5556072ed595-web-config\") pod \"alertmanager-main-0\" (UID: \"5138fe8c-9927-4d05-8fe5-5556072ed595\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:50:23.682524 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.682086    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:23.682524 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.682224    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7a6305de-8fd1-4d3e-9628-8c4cda07f379-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vrtnk\" (UID: \"7a6305de-8fd1-4d3e-9628-8c4cda07f379\") " pod="openshift-multus/multus-additional-cni-plugins-vrtnk"
Oct 30 21:50:23.682524 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.682252    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/a3459ebb-2fc2-4a82-b9c0-de5542d667e1-client-ca-bundle\") pod \"metrics-server-dc7c7d47d-ghdn7\" (UID: \"a3459ebb-2fc2-4a82-b9c0-de5542d667e1\") " pod="openshift-monitoring/metrics-server-dc7c7d47d-ghdn7"
Oct 30 21:50:23.682524 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.682254    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/87c44883-54d0-4a05-b94b-d3c2835a03e1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kn6xx\" (UID: \"87c44883-54d0-4a05-b94b-d3c2835a03e1\") " pod="openshift-insights/insights-runtime-extractor-kn6xx"
Oct 30 21:50:23.682524 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.682302    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0ad59239-edcc-46a4-a5a1-1debb63e3a5d-agent-certs\") pod \"konnectivity-agent-w72c9\" (UID: \"0ad59239-edcc-46a4-a5a1-1debb63e3a5d\") " pod="kube-system/konnectivity-agent-w72c9"
Oct 30 21:50:23.682524 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.682305    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fdf54bb9-f735-4810-82ee-d803099a1bcf-console-serving-cert\") pod \"console-7cbdbcf4b5-rmbsz\" (UID: \"fdf54bb9-f735-4810-82ee-d803099a1bcf\") " pod="openshift-console/console-7cbdbcf4b5-rmbsz"
Oct 30 21:50:23.682925 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.682574    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:23.682925 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.682582    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b954bab2-9dd7-4b39-bda6-3fec87381aff-original-pull-secret\") pod \"global-pull-secret-syncer-cck7d\" (UID: \"b954bab2-9dd7-4b39-bda6-3fec87381aff\") " pod="kube-system/global-pull-secret-syncer-cck7d"
Oct 30 21:50:23.682925 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.682603    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/49ff8cb6-54de-4e8c-9a35-634f9a1ef29b-metrics-tls\") pod \"dns-default-fdmdz\" (UID: \"49ff8cb6-54de-4e8c-9a35-634f9a1ef29b\") " pod="openshift-dns/dns-default-fdmdz"
Oct 30 21:50:23.682925 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.682638    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d947d3db-ab6e-4874-8817-f8c74be138a1-tmp\") pod \"tuned-22pl2\" (UID: \"d947d3db-ab6e-4874-8817-f8c74be138a1\") " pod="openshift-cluster-node-tuning-operator/tuned-22pl2"
Oct 30 21:50:23.682925 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.682636    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:23.682925 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.682704    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0ad59239-edcc-46a4-a5a1-1debb63e3a5d-konnectivity-ca\") pod \"konnectivity-agent-w72c9\" (UID: \"0ad59239-edcc-46a4-a5a1-1debb63e3a5d\") " pod="kube-system/konnectivity-agent-w72c9"
Oct 30 21:50:23.682925 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.682785    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/924394f7-5b3b-4767-a1ff-f1d310ece4ed-sys-fs\") pod \"aws-ebs-csi-driver-node-lrc2c\" (UID: \"924394f7-5b3b-4767-a1ff-f1d310ece4ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lrc2c"
Oct 30 21:50:23.682925 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.682843    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6980f40-89cf-4f3d-8cd5-91969ae5a276-metrics-certs\") pod \"network-metrics-daemon-xbh9m\" (UID: \"a6980f40-89cf-4f3d-8cd5-91969ae5a276\") " pod="openshift-multus/network-metrics-daemon-xbh9m"
Oct 30 21:50:23.683314 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.683097    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6980f40-89cf-4f3d-8cd5-91969ae5a276-metrics-certs\") pod \"network-metrics-daemon-xbh9m\" (UID: \"a6980f40-89cf-4f3d-8cd5-91969ae5a276\") " pod="openshift-multus/network-metrics-daemon-xbh9m"
Oct 30 21:50:23.684344 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.683591    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:23.684344 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.683756    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:23.684344 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.683953    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fdf54bb9-f735-4810-82ee-d803099a1bcf-console-serving-cert\") pod \"console-7cbdbcf4b5-rmbsz\" (UID: \"fdf54bb9-f735-4810-82ee-d803099a1bcf\") " pod="openshift-console/console-7cbdbcf4b5-rmbsz"
Oct 30 21:50:23.684344 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.683982    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0ad59239-edcc-46a4-a5a1-1debb63e3a5d-konnectivity-ca\") pod \"konnectivity-agent-w72c9\" (UID: \"0ad59239-edcc-46a4-a5a1-1debb63e3a5d\") " pod="kube-system/konnectivity-agent-w72c9"
Oct 30 21:50:23.684344 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.683980    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:23.684344 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.684082    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/dec1cd54-fec6-4c12-9d37-a1b0b1a7ffc0-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-pvl2l\" (UID: \"dec1cd54-fec6-4c12-9d37-a1b0b1a7ffc0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pvl2l"
Oct 30 21:50:23.684344 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.684322    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5f468f3c-bf3d-437c-8ec8-a5d56f448d99-multus-daemon-config\") pod \"multus-p7zpx\" (UID: \"5f468f3c-bf3d-437c-8ec8-a5d56f448d99\") " pod="openshift-multus/multus-p7zpx"
Oct 30 21:50:23.684344 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.684322    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/69e218ec-ab5e-4e09-8444-e53a710c53bf-service-ca\") pod \"console-69874d4cdd-zhlp5\" (UID: \"69e218ec-ab5e-4e09-8444-e53a710c53bf\") " pod="openshift-console/console-69874d4cdd-zhlp5"
Oct 30 21:50:23.684808 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.684363    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5138fe8c-9927-4d05-8fe5-5556072ed595-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5138fe8c-9927-4d05-8fe5-5556072ed595\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:50:23.708255 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.708235    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-920fbr0654gnu\""
Oct 30 21:50:23.712751 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.712730    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/a3459ebb-2fc2-4a82-b9c0-de5542d667e1-client-ca-bundle\") pod \"metrics-server-dc7c7d47d-ghdn7\" (UID: \"a3459ebb-2fc2-4a82-b9c0-de5542d667e1\") " pod="openshift-monitoring/metrics-server-dc7c7d47d-ghdn7"
Oct 30 21:50:23.715163 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.715117    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Oct 30 21:50:23.715569 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.715548    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Oct 30 21:50:23.715618 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.715550    5678 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 21:50:23.715659 ip-10-0-128-245 kubenswrapper[5678]: E1030 21:50:23.715644    5678 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-128-245.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-245.us-west-2.compute.internal"
Oct 30 21:50:23.716158 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.716136    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Oct 30 21:50:23.716246 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.716158    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Oct 30 21:50:23.720685 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.720665    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:23.721823 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.721800    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5138fe8c-9927-4d05-8fe5-5556072ed595-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5138fe8c-9927-4d05-8fe5-5556072ed595\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:50:23.725016 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.724997    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fdf54bb9-f735-4810-82ee-d803099a1bcf-trusted-ca-bundle\") pod \"console-7cbdbcf4b5-rmbsz\" (UID: \"fdf54bb9-f735-4810-82ee-d803099a1bcf\") " pod="openshift-console/console-7cbdbcf4b5-rmbsz"
Oct 30 21:50:23.728320 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.728301    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgn7x\" (UniqueName: \"kubernetes.io/projected/5f468f3c-bf3d-437c-8ec8-a5d56f448d99-kube-api-access-rgn7x\") pod \"multus-p7zpx\" (UID: \"5f468f3c-bf3d-437c-8ec8-a5d56f448d99\") " pod="openshift-multus/multus-p7zpx"
Oct 30 21:50:23.729182 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.729098    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:23.742781 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.739209    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jt5t\" (UniqueName: \"kubernetes.io/projected/f938a93b-29dd-4e7d-b8e9-444df68acf03-kube-api-access-6jt5t\") pod \"node-exporter-7j5j6\" (UID: \"f938a93b-29dd-4e7d-b8e9-444df68acf03\") " pod="openshift-monitoring/node-exporter-7j5j6"
Oct 30 21:50:23.742781 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.740020    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvbdq\" (UniqueName: \"kubernetes.io/projected/dec1cd54-fec6-4c12-9d37-a1b0b1a7ffc0-kube-api-access-wvbdq\") pod \"kube-state-metrics-5dc8d7d89b-pvl2l\" (UID: \"dec1cd54-fec6-4c12-9d37-a1b0b1a7ffc0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pvl2l"
Oct 30 21:50:23.742781 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.740805    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5jdv\" (UniqueName: \"kubernetes.io/projected/fdf54bb9-f735-4810-82ee-d803099a1bcf-kube-api-access-s5jdv\") pod \"console-7cbdbcf4b5-rmbsz\" (UID: \"fdf54bb9-f735-4810-82ee-d803099a1bcf\") " pod="openshift-console/console-7cbdbcf4b5-rmbsz"
Oct 30 21:50:23.742781 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.741296    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7qxb\" (UniqueName: \"kubernetes.io/projected/49ff8cb6-54de-4e8c-9a35-634f9a1ef29b-kube-api-access-z7qxb\") pod \"dns-default-fdmdz\" (UID: \"49ff8cb6-54de-4e8c-9a35-634f9a1ef29b\") " pod="openshift-dns/dns-default-fdmdz"
Oct 30 21:50:23.743942 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.743907    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6x28\" (UniqueName: \"kubernetes.io/projected/55619a2e-f6e9-4719-b362-30b69d553967-kube-api-access-j6x28\") pod \"network-check-target-w9tx8\" (UID: \"55619a2e-f6e9-4719-b362-30b69d553967\") " pod="openshift-network-diagnostics/network-check-target-w9tx8"
Oct 30 21:50:23.744150 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.744132    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fn582\" (UniqueName: \"kubernetes.io/projected/67b50e2a-4d01-4d5f-a5d1-7e80ee62642a-kube-api-access-fn582\") pod \"prometheus-operator-9cd89b74c-vcj4q\" (UID: \"67b50e2a-4d01-4d5f-a5d1-7e80ee62642a\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-vcj4q"
Oct 30 21:50:23.744297 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.744275    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hghq\" (UniqueName: \"kubernetes.io/projected/41fe6d63-9316-4916-b042-87ba7e602167-kube-api-access-2hghq\") pod \"ovnkube-node-ml9xf\" (UID: \"41fe6d63-9316-4916-b042-87ba7e602167\") " pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf"
Oct 30 21:50:23.744436 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.744393    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6v9rf\" (UniqueName: \"kubernetes.io/projected/5138fe8c-9927-4d05-8fe5-5556072ed595-kube-api-access-6v9rf\") pod \"alertmanager-main-0\" (UID: \"5138fe8c-9927-4d05-8fe5-5556072ed595\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:50:23.744610 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.744590    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnt2l\" (UniqueName: \"kubernetes.io/projected/924394f7-5b3b-4767-a1ff-f1d310ece4ed-kube-api-access-qnt2l\") pod \"aws-ebs-csi-driver-node-lrc2c\" (UID: \"924394f7-5b3b-4767-a1ff-f1d310ece4ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lrc2c"
Oct 30 21:50:23.744667 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.744640    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdznq\" (UniqueName: \"kubernetes.io/projected/87c44883-54d0-4a05-b94b-d3c2835a03e1-kube-api-access-cdznq\") pod \"insights-runtime-extractor-kn6xx\" (UID: \"87c44883-54d0-4a05-b94b-d3c2835a03e1\") " pod="openshift-insights/insights-runtime-extractor-kn6xx"
Oct 30 21:50:23.744940 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.744925    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gv6wf\" (UniqueName: \"kubernetes.io/projected/69e218ec-ab5e-4e09-8444-e53a710c53bf-kube-api-access-gv6wf\") pod \"console-69874d4cdd-zhlp5\" (UID: \"69e218ec-ab5e-4e09-8444-e53a710c53bf\") " pod="openshift-console/console-69874d4cdd-zhlp5"
Oct 30 21:50:23.745078 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.745058    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cr8x2\" (UniqueName: \"kubernetes.io/projected/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-kube-api-access-cr8x2\") pod \"prometheus-k8s-0\" (UID: \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:23.745130 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.745115    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkkmr\" (UniqueName: \"kubernetes.io/projected/d947d3db-ab6e-4874-8817-f8c74be138a1-kube-api-access-mkkmr\") pod \"tuned-22pl2\" (UID: \"d947d3db-ab6e-4874-8817-f8c74be138a1\") " pod="openshift-cluster-node-tuning-operator/tuned-22pl2"
Oct 30 21:50:23.745213 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.745193    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czfxj\" (UniqueName: \"kubernetes.io/projected/a6980f40-89cf-4f3d-8cd5-91969ae5a276-kube-api-access-czfxj\") pod \"network-metrics-daemon-xbh9m\" (UID: \"a6980f40-89cf-4f3d-8cd5-91969ae5a276\") " pod="openshift-multus/network-metrics-daemon-xbh9m"
Oct 30 21:50:23.745278 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.745265    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8w8vm\" (UniqueName: \"kubernetes.io/projected/f22e4514-ac6d-481d-8afe-6e2e4cdc3aa1-kube-api-access-8w8vm\") pod \"iptables-alerter-t24lz\" (UID: \"f22e4514-ac6d-481d-8afe-6e2e4cdc3aa1\") " pod="openshift-network-operator/iptables-alerter-t24lz"
Oct 30 21:50:23.745638 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.745619    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b97qt\" (UniqueName: \"kubernetes.io/projected/82f75dd5-40cd-4cb3-9111-6213d6b43679-kube-api-access-b97qt\") pod \"node-ca-gmbbw\" (UID: \"82f75dd5-40cd-4cb3-9111-6213d6b43679\") " pod="openshift-image-registry/node-ca-gmbbw"
Oct 30 21:50:23.745707 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.745693    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxd2f\" (UniqueName: \"kubernetes.io/projected/32d63b18-0fcb-4e4a-a24c-ee044dcb5a90-kube-api-access-pxd2f\") pod \"node-resolver-gbpvx\" (UID: \"32d63b18-0fcb-4e4a-a24c-ee044dcb5a90\") " pod="openshift-dns/node-resolver-gbpvx"
Oct 30 21:50:23.746001 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.745982    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hdbq\" (UniqueName: \"kubernetes.io/projected/a3459ebb-2fc2-4a82-b9c0-de5542d667e1-kube-api-access-6hdbq\") pod \"metrics-server-dc7c7d47d-ghdn7\" (UID: \"a3459ebb-2fc2-4a82-b9c0-de5542d667e1\") " pod="openshift-monitoring/metrics-server-dc7c7d47d-ghdn7"
Oct 30 21:50:23.749755 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.749734    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwvcm\" (UniqueName: \"kubernetes.io/projected/7a6305de-8fd1-4d3e-9628-8c4cda07f379-kube-api-access-lwvcm\") pod \"multus-additional-cni-plugins-vrtnk\" (UID: \"7a6305de-8fd1-4d3e-9628-8c4cda07f379\") " pod="openshift-multus/multus-additional-cni-plugins-vrtnk"
Oct 30 21:50:23.749868 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.749847    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66wt8\" (UniqueName: \"kubernetes.io/projected/516573f9-4ada-4868-b885-f61b9640e6bf-kube-api-access-66wt8\") pod \"ingress-canary-hr227\" (UID: \"516573f9-4ada-4868-b885-f61b9640e6bf\") " pod="openshift-ingress-canary/ingress-canary-hr227"
Oct 30 21:50:23.789760 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.789730    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5f468f3c-bf3d-437c-8ec8-a5d56f448d99-host-var-lib-cni-multus\") pod \"multus-p7zpx\" (UID: \"5f468f3c-bf3d-437c-8ec8-a5d56f448d99\") " pod="openshift-multus/multus-p7zpx"
Oct 30 21:50:23.789903 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.789766    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f938a93b-29dd-4e7d-b8e9-444df68acf03-sys\") pod \"node-exporter-7j5j6\" (UID: \"f938a93b-29dd-4e7d-b8e9-444df68acf03\") " pod="openshift-monitoring/node-exporter-7j5j6"
Oct 30 21:50:23.789903 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.789791    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f22e4514-ac6d-481d-8afe-6e2e4cdc3aa1-host-slash\") pod \"iptables-alerter-t24lz\" (UID: \"f22e4514-ac6d-481d-8afe-6e2e4cdc3aa1\") " pod="openshift-network-operator/iptables-alerter-t24lz"
Oct 30 21:50:23.789903 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.789815    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5f468f3c-bf3d-437c-8ec8-a5d56f448d99-multus-cni-dir\") pod \"multus-p7zpx\" (UID: \"5f468f3c-bf3d-437c-8ec8-a5d56f448d99\") " pod="openshift-multus/multus-p7zpx"
Oct 30 21:50:23.789903 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.789837    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/41fe6d63-9316-4916-b042-87ba7e602167-run-ovn\") pod \"ovnkube-node-ml9xf\" (UID: \"41fe6d63-9316-4916-b042-87ba7e602167\") " pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf"
Oct 30 21:50:23.789903 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.789861    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5f468f3c-bf3d-437c-8ec8-a5d56f448d99-host-run-k8s-cni-cncf-io\") pod \"multus-p7zpx\" (UID: \"5f468f3c-bf3d-437c-8ec8-a5d56f448d99\") " pod="openshift-multus/multus-p7zpx"
Oct 30 21:50:23.789903 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.789882    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/82f75dd5-40cd-4cb3-9111-6213d6b43679-host\") pod \"node-ca-gmbbw\" (UID: \"82f75dd5-40cd-4cb3-9111-6213d6b43679\") " pod="openshift-image-registry/node-ca-gmbbw"
Oct 30 21:50:23.790201 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.789905    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/41fe6d63-9316-4916-b042-87ba7e602167-var-lib-openvswitch\") pod \"ovnkube-node-ml9xf\" (UID: \"41fe6d63-9316-4916-b042-87ba7e602167\") " pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf"
Oct 30 21:50:23.790201 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.789930    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d947d3db-ab6e-4874-8817-f8c74be138a1-etc-sysctl-conf\") pod \"tuned-22pl2\" (UID: \"d947d3db-ab6e-4874-8817-f8c74be138a1\") " pod="openshift-cluster-node-tuning-operator/tuned-22pl2"
Oct 30 21:50:23.790201 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.789953    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d947d3db-ab6e-4874-8817-f8c74be138a1-host\") pod \"tuned-22pl2\" (UID: \"d947d3db-ab6e-4874-8817-f8c74be138a1\") " pod="openshift-cluster-node-tuning-operator/tuned-22pl2"
Oct 30 21:50:23.790201 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.789985    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/924394f7-5b3b-4767-a1ff-f1d310ece4ed-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lrc2c\" (UID: \"924394f7-5b3b-4767-a1ff-f1d310ece4ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lrc2c"
Oct 30 21:50:23.790201 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.790014    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/41fe6d63-9316-4916-b042-87ba7e602167-host-kubelet\") pod \"ovnkube-node-ml9xf\" (UID: \"41fe6d63-9316-4916-b042-87ba7e602167\") " pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf"
Oct 30 21:50:23.790201 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.790038    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d947d3db-ab6e-4874-8817-f8c74be138a1-etc-kubernetes\") pod \"tuned-22pl2\" (UID: \"d947d3db-ab6e-4874-8817-f8c74be138a1\") " pod="openshift-cluster-node-tuning-operator/tuned-22pl2"
Oct 30 21:50:23.790201 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.790061    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d947d3db-ab6e-4874-8817-f8c74be138a1-sys\") pod \"tuned-22pl2\" (UID: \"d947d3db-ab6e-4874-8817-f8c74be138a1\") " pod="openshift-cluster-node-tuning-operator/tuned-22pl2"
Oct 30 21:50:23.790201 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.790084    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b954bab2-9dd7-4b39-bda6-3fec87381aff-kubelet-config\") pod \"global-pull-secret-syncer-cck7d\" (UID: \"b954bab2-9dd7-4b39-bda6-3fec87381aff\") " pod="kube-system/global-pull-secret-syncer-cck7d"
Oct 30 21:50:23.790201 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.790107    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/41fe6d63-9316-4916-b042-87ba7e602167-host-cni-bin\") pod \"ovnkube-node-ml9xf\" (UID: \"41fe6d63-9316-4916-b042-87ba7e602167\") " pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf"
Oct 30 21:50:23.790201 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.790134    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5f468f3c-bf3d-437c-8ec8-a5d56f448d99-hostroot\") pod \"multus-p7zpx\" (UID: \"5f468f3c-bf3d-437c-8ec8-a5d56f448d99\") " pod="openshift-multus/multus-p7zpx"
Oct 30 21:50:23.790201 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.790157    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/41fe6d63-9316-4916-b042-87ba7e602167-log-socket\") pod \"ovnkube-node-ml9xf\" (UID: \"41fe6d63-9316-4916-b042-87ba7e602167\") " pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf"
Oct 30 21:50:23.790201 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.790183    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7a6305de-8fd1-4d3e-9628-8c4cda07f379-system-cni-dir\") pod \"multus-additional-cni-plugins-vrtnk\" (UID: \"7a6305de-8fd1-4d3e-9628-8c4cda07f379\") " pod="openshift-multus/multus-additional-cni-plugins-vrtnk"
Oct 30 21:50:23.790820 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.790211    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/924394f7-5b3b-4767-a1ff-f1d310ece4ed-socket-dir\") pod \"aws-ebs-csi-driver-node-lrc2c\" (UID: \"924394f7-5b3b-4767-a1ff-f1d310ece4ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lrc2c"
Oct 30 21:50:23.790820 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.790237    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f938a93b-29dd-4e7d-b8e9-444df68acf03-root\") pod \"node-exporter-7j5j6\" (UID: \"f938a93b-29dd-4e7d-b8e9-444df68acf03\") " pod="openshift-monitoring/node-exporter-7j5j6"
Oct 30 21:50:23.790820 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.790267    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5f468f3c-bf3d-437c-8ec8-a5d56f448d99-host-var-lib-kubelet\") pod \"multus-p7zpx\" (UID: \"5f468f3c-bf3d-437c-8ec8-a5d56f448d99\") " pod="openshift-multus/multus-p7zpx"
Oct 30 21:50:23.790820 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.790292    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/41fe6d63-9316-4916-b042-87ba7e602167-etc-openvswitch\") pod \"ovnkube-node-ml9xf\" (UID: \"41fe6d63-9316-4916-b042-87ba7e602167\") " pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf"
Oct 30 21:50:23.790820 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.790317    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/87c44883-54d0-4a05-b94b-d3c2835a03e1-crio-socket\") pod \"insights-runtime-extractor-kn6xx\" (UID: \"87c44883-54d0-4a05-b94b-d3c2835a03e1\") " pod="openshift-insights/insights-runtime-extractor-kn6xx"
Oct 30 21:50:23.790820 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.790343    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/924394f7-5b3b-4767-a1ff-f1d310ece4ed-registration-dir\") pod \"aws-ebs-csi-driver-node-lrc2c\" (UID: \"924394f7-5b3b-4767-a1ff-f1d310ece4ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lrc2c"
Oct 30 21:50:23.790820 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.790368    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5f468f3c-bf3d-437c-8ec8-a5d56f448d99-host-run-netns\") pod \"multus-p7zpx\" (UID: \"5f468f3c-bf3d-437c-8ec8-a5d56f448d99\") " pod="openshift-multus/multus-p7zpx"
Oct 30 21:50:23.790820 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.790394    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/41fe6d63-9316-4916-b042-87ba7e602167-host-run-ovn-kubernetes\") pod \"ovnkube-node-ml9xf\" (UID: \"41fe6d63-9316-4916-b042-87ba7e602167\") " pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf"
Oct 30 21:50:23.790820 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.790436    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/32d63b18-0fcb-4e4a-a24c-ee044dcb5a90-hosts-file\") pod \"node-resolver-gbpvx\" (UID: \"32d63b18-0fcb-4e4a-a24c-ee044dcb5a90\") " pod="openshift-dns/node-resolver-gbpvx"
Oct 30 21:50:23.790820 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.790464    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d947d3db-ab6e-4874-8817-f8c74be138a1-lib-modules\") pod \"tuned-22pl2\" (UID: \"d947d3db-ab6e-4874-8817-f8c74be138a1\") " pod="openshift-cluster-node-tuning-operator/tuned-22pl2"
Oct 30 21:50:23.790820 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.790490    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d947d3db-ab6e-4874-8817-f8c74be138a1-var-lib-kubelet\") pod \"tuned-22pl2\" (UID: \"d947d3db-ab6e-4874-8817-f8c74be138a1\") " pod="openshift-cluster-node-tuning-operator/tuned-22pl2"
Oct 30 21:50:23.790820 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.790505    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d947d3db-ab6e-4874-8817-f8c74be138a1-etc-kubernetes\") pod \"tuned-22pl2\" (UID: \"d947d3db-ab6e-4874-8817-f8c74be138a1\") " pod="openshift-cluster-node-tuning-operator/tuned-22pl2"
Oct 30 21:50:23.790820 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.790511    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5f468f3c-bf3d-437c-8ec8-a5d56f448d99-host-var-lib-cni-multus\") pod \"multus-p7zpx\" (UID: \"5f468f3c-bf3d-437c-8ec8-a5d56f448d99\") " pod="openshift-multus/multus-p7zpx"
Oct 30 21:50:23.790820 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.790516    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/41fe6d63-9316-4916-b042-87ba7e602167-host-slash\") pod \"ovnkube-node-ml9xf\" (UID: \"41fe6d63-9316-4916-b042-87ba7e602167\") " pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf"
Oct 30 21:50:23.790820 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.790555    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/41fe6d63-9316-4916-b042-87ba7e602167-host-slash\") pod \"ovnkube-node-ml9xf\" (UID: \"41fe6d63-9316-4916-b042-87ba7e602167\") " pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf"
Oct 30 21:50:23.790820 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.790576    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f938a93b-29dd-4e7d-b8e9-444df68acf03-sys\") pod \"node-exporter-7j5j6\" (UID: \"f938a93b-29dd-4e7d-b8e9-444df68acf03\") " pod="openshift-monitoring/node-exporter-7j5j6"
Oct 30 21:50:23.790820 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.790579    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/41fe6d63-9316-4916-b042-87ba7e602167-host-run-netns\") pod \"ovnkube-node-ml9xf\" (UID: \"41fe6d63-9316-4916-b042-87ba7e602167\") " pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf"
Oct 30 21:50:23.790820 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.790609    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/41fe6d63-9316-4916-b042-87ba7e602167-host-run-netns\") pod \"ovnkube-node-ml9xf\" (UID: \"41fe6d63-9316-4916-b042-87ba7e602167\") " pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf"
Oct 30 21:50:23.790820 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.790610    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d947d3db-ab6e-4874-8817-f8c74be138a1-sys\") pod \"tuned-22pl2\" (UID: \"d947d3db-ab6e-4874-8817-f8c74be138a1\") " pod="openshift-cluster-node-tuning-operator/tuned-22pl2"
Oct 30 21:50:23.790820 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.790619    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/41fe6d63-9316-4916-b042-87ba7e602167-run-systemd\") pod \"ovnkube-node-ml9xf\" (UID: \"41fe6d63-9316-4916-b042-87ba7e602167\") " pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf"
Oct 30 21:50:23.790820 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.790648    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/924394f7-5b3b-4767-a1ff-f1d310ece4ed-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lrc2c\" (UID: \"924394f7-5b3b-4767-a1ff-f1d310ece4ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lrc2c"
Oct 30 21:50:23.790820 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.790649    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/82f75dd5-40cd-4cb3-9111-6213d6b43679-host\") pod \"node-ca-gmbbw\" (UID: \"82f75dd5-40cd-4cb3-9111-6213d6b43679\") " pod="openshift-image-registry/node-ca-gmbbw"
Oct 30 21:50:23.790820 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.790657    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/924394f7-5b3b-4767-a1ff-f1d310ece4ed-etc-selinux\") pod \"aws-ebs-csi-driver-node-lrc2c\" (UID: \"924394f7-5b3b-4767-a1ff-f1d310ece4ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lrc2c"
Oct 30 21:50:23.790820 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.790658    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d947d3db-ab6e-4874-8817-f8c74be138a1-host\") pod \"tuned-22pl2\" (UID: \"d947d3db-ab6e-4874-8817-f8c74be138a1\") " pod="openshift-cluster-node-tuning-operator/tuned-22pl2"
Oct 30 21:50:23.790820 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.790673    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/41fe6d63-9316-4916-b042-87ba7e602167-host-kubelet\") pod \"ovnkube-node-ml9xf\" (UID: \"41fe6d63-9316-4916-b042-87ba7e602167\") " pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf"
Oct 30 21:50:23.790820 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.790697    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5f468f3c-bf3d-437c-8ec8-a5d56f448d99-multus-socket-dir-parent\") pod \"multus-p7zpx\" (UID: \"5f468f3c-bf3d-437c-8ec8-a5d56f448d99\") " pod="openshift-multus/multus-p7zpx"
Oct 30 21:50:23.790820 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.790700    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/924394f7-5b3b-4767-a1ff-f1d310ece4ed-etc-selinux\") pod \"aws-ebs-csi-driver-node-lrc2c\" (UID: \"924394f7-5b3b-4767-a1ff-f1d310ece4ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lrc2c"
Oct 30 21:50:23.790820 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.790723    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/87c44883-54d0-4a05-b94b-d3c2835a03e1-crio-socket\") pod \"insights-runtime-extractor-kn6xx\" (UID: \"87c44883-54d0-4a05-b94b-d3c2835a03e1\") " pod="openshift-insights/insights-runtime-extractor-kn6xx"
Oct 30 21:50:23.790820 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.790732    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5f468f3c-bf3d-437c-8ec8-a5d56f448d99-multus-socket-dir-parent\") pod \"multus-p7zpx\" (UID: \"5f468f3c-bf3d-437c-8ec8-a5d56f448d99\") " pod="openshift-multus/multus-p7zpx"
Oct 30 21:50:23.790820 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.790731    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b954bab2-9dd7-4b39-bda6-3fec87381aff-kubelet-config\") pod \"global-pull-secret-syncer-cck7d\" (UID: \"b954bab2-9dd7-4b39-bda6-3fec87381aff\") " pod="kube-system/global-pull-secret-syncer-cck7d"
Oct 30 21:50:23.790820 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.790747    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f22e4514-ac6d-481d-8afe-6e2e4cdc3aa1-host-slash\") pod \"iptables-alerter-t24lz\" (UID: \"f22e4514-ac6d-481d-8afe-6e2e4cdc3aa1\") " pod="openshift-network-operator/iptables-alerter-t24lz"
Oct 30 21:50:23.790820 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.790750    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/41fe6d63-9316-4916-b042-87ba7e602167-node-log\") pod \"ovnkube-node-ml9xf\" (UID: \"41fe6d63-9316-4916-b042-87ba7e602167\") " pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf"
Oct 30 21:50:23.790820 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.790764    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/41fe6d63-9316-4916-b042-87ba7e602167-var-lib-openvswitch\") pod \"ovnkube-node-ml9xf\" (UID: \"41fe6d63-9316-4916-b042-87ba7e602167\") " pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf"
Oct 30 21:50:23.790820 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.790724    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/41fe6d63-9316-4916-b042-87ba7e602167-node-log\") pod \"ovnkube-node-ml9xf\" (UID: \"41fe6d63-9316-4916-b042-87ba7e602167\") " pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf"
Oct 30 21:50:23.790820 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.790775    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/41fe6d63-9316-4916-b042-87ba7e602167-run-ovn\") pod \"ovnkube-node-ml9xf\" (UID: \"41fe6d63-9316-4916-b042-87ba7e602167\") " pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf"
Oct 30 21:50:23.790820 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.790605    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d947d3db-ab6e-4874-8817-f8c74be138a1-etc-sysctl-conf\") pod \"tuned-22pl2\" (UID: \"d947d3db-ab6e-4874-8817-f8c74be138a1\") " pod="openshift-cluster-node-tuning-operator/tuned-22pl2"
Oct 30 21:50:23.790820 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.790773    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/924394f7-5b3b-4767-a1ff-f1d310ece4ed-registration-dir\") pod \"aws-ebs-csi-driver-node-lrc2c\" (UID: \"924394f7-5b3b-4767-a1ff-f1d310ece4ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lrc2c"
Oct 30 21:50:23.790820 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.790806    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5f468f3c-bf3d-437c-8ec8-a5d56f448d99-multus-cni-dir\") pod \"multus-p7zpx\" (UID: \"5f468f3c-bf3d-437c-8ec8-a5d56f448d99\") " pod="openshift-multus/multus-p7zpx"
Oct 30 21:50:23.790820 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.790813    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/41fe6d63-9316-4916-b042-87ba7e602167-etc-openvswitch\") pod \"ovnkube-node-ml9xf\" (UID: \"41fe6d63-9316-4916-b042-87ba7e602167\") " pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf"
Oct 30 21:50:23.790820 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.790817    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5f468f3c-bf3d-437c-8ec8-a5d56f448d99-host-run-k8s-cni-cncf-io\") pod \"multus-p7zpx\" (UID: \"5f468f3c-bf3d-437c-8ec8-a5d56f448d99\") " pod="openshift-multus/multus-p7zpx"
Oct 30 21:50:23.790820 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.790835    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7a6305de-8fd1-4d3e-9628-8c4cda07f379-system-cni-dir\") pod \"multus-additional-cni-plugins-vrtnk\" (UID: \"7a6305de-8fd1-4d3e-9628-8c4cda07f379\") " pod="openshift-multus/multus-additional-cni-plugins-vrtnk"
Oct 30 21:50:23.790820 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.790844    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/924394f7-5b3b-4767-a1ff-f1d310ece4ed-socket-dir\") pod \"aws-ebs-csi-driver-node-lrc2c\" (UID: \"924394f7-5b3b-4767-a1ff-f1d310ece4ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lrc2c"
Oct 30 21:50:23.790820 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.790847    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7a6305de-8fd1-4d3e-9628-8c4cda07f379-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vrtnk\" (UID: \"7a6305de-8fd1-4d3e-9628-8c4cda07f379\") " pod="openshift-multus/multus-additional-cni-plugins-vrtnk"
Oct 30 21:50:23.792369 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.790859    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/41fe6d63-9316-4916-b042-87ba7e602167-host-cni-bin\") pod \"ovnkube-node-ml9xf\" (UID: \"41fe6d63-9316-4916-b042-87ba7e602167\") " pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf"
Oct 30 21:50:23.792369 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.790867    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f938a93b-29dd-4e7d-b8e9-444df68acf03-root\") pod \"node-exporter-7j5j6\" (UID: \"f938a93b-29dd-4e7d-b8e9-444df68acf03\") " pod="openshift-monitoring/node-exporter-7j5j6"
Oct 30 21:50:23.792369 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.790885    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/32d63b18-0fcb-4e4a-a24c-ee044dcb5a90-hosts-file\") pod \"node-resolver-gbpvx\" (UID: \"32d63b18-0fcb-4e4a-a24c-ee044dcb5a90\") " pod="openshift-dns/node-resolver-gbpvx"
Oct 30 21:50:23.792369 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.790891    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5f468f3c-bf3d-437c-8ec8-a5d56f448d99-host-run-netns\") pod \"multus-p7zpx\" (UID: \"5f468f3c-bf3d-437c-8ec8-a5d56f448d99\") " pod="openshift-multus/multus-p7zpx"
Oct 30 21:50:23.792369 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.790894    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5f468f3c-bf3d-437c-8ec8-a5d56f448d99-host-var-lib-kubelet\") pod \"multus-p7zpx\" (UID: \"5f468f3c-bf3d-437c-8ec8-a5d56f448d99\") " pod="openshift-multus/multus-p7zpx"
Oct 30 21:50:23.792369 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.790906    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/41fe6d63-9316-4916-b042-87ba7e602167-log-socket\") pod \"ovnkube-node-ml9xf\" (UID: \"41fe6d63-9316-4916-b042-87ba7e602167\") " pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf"
Oct 30 21:50:23.792369 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.790908    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5f468f3c-bf3d-437c-8ec8-a5d56f448d99-hostroot\") pod \"multus-p7zpx\" (UID: \"5f468f3c-bf3d-437c-8ec8-a5d56f448d99\") " pod="openshift-multus/multus-p7zpx"
Oct 30 21:50:23.792369 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.790917    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d947d3db-ab6e-4874-8817-f8c74be138a1-var-lib-kubelet\") pod \"tuned-22pl2\" (UID: \"d947d3db-ab6e-4874-8817-f8c74be138a1\") " pod="openshift-cluster-node-tuning-operator/tuned-22pl2"
Oct 30 21:50:23.792369 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.790932    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/41fe6d63-9316-4916-b042-87ba7e602167-host-run-ovn-kubernetes\") pod \"ovnkube-node-ml9xf\" (UID: \"41fe6d63-9316-4916-b042-87ba7e602167\") " pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf"
Oct 30 21:50:23.792369 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.790935    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/41fe6d63-9316-4916-b042-87ba7e602167-run-systemd\") pod \"ovnkube-node-ml9xf\" (UID: \"41fe6d63-9316-4916-b042-87ba7e602167\") " pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf"
Oct 30 21:50:23.792369 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.790948    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7a6305de-8fd1-4d3e-9628-8c4cda07f379-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vrtnk\" (UID: \"7a6305de-8fd1-4d3e-9628-8c4cda07f379\") " pod="openshift-multus/multus-additional-cni-plugins-vrtnk"
Oct 30 21:50:23.792369 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.790974    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d947d3db-ab6e-4874-8817-f8c74be138a1-lib-modules\") pod \"tuned-22pl2\" (UID: \"d947d3db-ab6e-4874-8817-f8c74be138a1\") " pod="openshift-cluster-node-tuning-operator/tuned-22pl2"
Oct 30 21:50:23.792369 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.790988    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5f468f3c-bf3d-437c-8ec8-a5d56f448d99-etc-kubernetes\") pod \"multus-p7zpx\" (UID: \"5f468f3c-bf3d-437c-8ec8-a5d56f448d99\") " pod="openshift-multus/multus-p7zpx"
Oct 30 21:50:23.792369 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.791026    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5f468f3c-bf3d-437c-8ec8-a5d56f448d99-etc-kubernetes\") pod \"multus-p7zpx\" (UID: \"5f468f3c-bf3d-437c-8ec8-a5d56f448d99\") " pod="openshift-multus/multus-p7zpx"
Oct 30 21:50:23.792369 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.791052    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b954bab2-9dd7-4b39-bda6-3fec87381aff-dbus\") pod \"global-pull-secret-syncer-cck7d\" (UID: \"b954bab2-9dd7-4b39-bda6-3fec87381aff\") " pod="kube-system/global-pull-secret-syncer-cck7d"
Oct 30 21:50:23.792369 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.791075    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/41fe6d63-9316-4916-b042-87ba7e602167-systemd-units\") pod \"ovnkube-node-ml9xf\" (UID: \"41fe6d63-9316-4916-b042-87ba7e602167\") " pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf"
Oct 30 21:50:23.792369 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.791099    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/41fe6d63-9316-4916-b042-87ba7e602167-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ml9xf\" (UID: \"41fe6d63-9316-4916-b042-87ba7e602167\") " pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf"
Oct 30 21:50:23.792369 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.791130    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/41fe6d63-9316-4916-b042-87ba7e602167-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ml9xf\" (UID: \"41fe6d63-9316-4916-b042-87ba7e602167\") " pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf"
Oct 30 21:50:23.792369 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.791154    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b954bab2-9dd7-4b39-bda6-3fec87381aff-dbus\") pod \"global-pull-secret-syncer-cck7d\" (UID: \"b954bab2-9dd7-4b39-bda6-3fec87381aff\") " pod="kube-system/global-pull-secret-syncer-cck7d"
Oct 30 21:50:23.792369 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.791165    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/41fe6d63-9316-4916-b042-87ba7e602167-systemd-units\") pod \"ovnkube-node-ml9xf\" (UID: \"41fe6d63-9316-4916-b042-87ba7e602167\") " pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf"
Oct 30 21:50:23.792369 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.791193    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d947d3db-ab6e-4874-8817-f8c74be138a1-etc-sysconfig\") pod \"tuned-22pl2\" (UID: \"d947d3db-ab6e-4874-8817-f8c74be138a1\") " pod="openshift-cluster-node-tuning-operator/tuned-22pl2"
Oct 30 21:50:23.792369 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.791260    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d947d3db-ab6e-4874-8817-f8c74be138a1-etc-sysconfig\") pod \"tuned-22pl2\" (UID: \"d947d3db-ab6e-4874-8817-f8c74be138a1\") " pod="openshift-cluster-node-tuning-operator/tuned-22pl2"
Oct 30 21:50:23.792369 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.791262    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/924394f7-5b3b-4767-a1ff-f1d310ece4ed-sys-fs\") pod \"aws-ebs-csi-driver-node-lrc2c\" (UID: \"924394f7-5b3b-4767-a1ff-f1d310ece4ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lrc2c"
Oct 30 21:50:23.792369 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.791290    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/41fe6d63-9316-4916-b042-87ba7e602167-host-cni-netd\") pod \"ovnkube-node-ml9xf\" (UID: \"41fe6d63-9316-4916-b042-87ba7e602167\") " pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf"
Oct 30 21:50:23.792369 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.791308    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d947d3db-ab6e-4874-8817-f8c74be138a1-etc-systemd\") pod \"tuned-22pl2\" (UID: \"d947d3db-ab6e-4874-8817-f8c74be138a1\") " pod="openshift-cluster-node-tuning-operator/tuned-22pl2"
Oct 30 21:50:23.792369 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.791312    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/924394f7-5b3b-4767-a1ff-f1d310ece4ed-sys-fs\") pod \"aws-ebs-csi-driver-node-lrc2c\" (UID: \"924394f7-5b3b-4767-a1ff-f1d310ece4ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lrc2c"
Oct 30 21:50:23.792369 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.791325    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7a6305de-8fd1-4d3e-9628-8c4cda07f379-cnibin\") pod \"multus-additional-cni-plugins-vrtnk\" (UID: \"7a6305de-8fd1-4d3e-9628-8c4cda07f379\") " pod="openshift-multus/multus-additional-cni-plugins-vrtnk"
Oct 30 21:50:23.792369 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.791340    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5f468f3c-bf3d-437c-8ec8-a5d56f448d99-cnibin\") pod \"multus-p7zpx\" (UID: \"5f468f3c-bf3d-437c-8ec8-a5d56f448d99\") " pod="openshift-multus/multus-p7zpx"
Oct 30 21:50:23.792369 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.791351    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/41fe6d63-9316-4916-b042-87ba7e602167-host-cni-netd\") pod \"ovnkube-node-ml9xf\" (UID: \"41fe6d63-9316-4916-b042-87ba7e602167\") " pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf"
Oct 30 21:50:23.792369 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.791355    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5f468f3c-bf3d-437c-8ec8-a5d56f448d99-host-run-multus-certs\") pod \"multus-p7zpx\" (UID: \"5f468f3c-bf3d-437c-8ec8-a5d56f448d99\") " pod="openshift-multus/multus-p7zpx"
Oct 30 21:50:23.792369 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.791381    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5f468f3c-bf3d-437c-8ec8-a5d56f448d99-host-run-multus-certs\") pod \"multus-p7zpx\" (UID: \"5f468f3c-bf3d-437c-8ec8-a5d56f448d99\") " pod="openshift-multus/multus-p7zpx"
Oct 30 21:50:23.792369 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.791385    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7a6305de-8fd1-4d3e-9628-8c4cda07f379-cnibin\") pod \"multus-additional-cni-plugins-vrtnk\" (UID: \"7a6305de-8fd1-4d3e-9628-8c4cda07f379\") " pod="openshift-multus/multus-additional-cni-plugins-vrtnk"
Oct 30 21:50:23.792369 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.791388    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d947d3db-ab6e-4874-8817-f8c74be138a1-etc-systemd\") pod \"tuned-22pl2\" (UID: \"d947d3db-ab6e-4874-8817-f8c74be138a1\") " pod="openshift-cluster-node-tuning-operator/tuned-22pl2"
Oct 30 21:50:23.792369 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.791510    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5f468f3c-bf3d-437c-8ec8-a5d56f448d99-cnibin\") pod \"multus-p7zpx\" (UID: \"5f468f3c-bf3d-437c-8ec8-a5d56f448d99\") " pod="openshift-multus/multus-p7zpx"
Oct 30 21:50:23.792369 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.791516    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/41fe6d63-9316-4916-b042-87ba7e602167-run-openvswitch\") pod \"ovnkube-node-ml9xf\" (UID: \"41fe6d63-9316-4916-b042-87ba7e602167\") " pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf"
Oct 30 21:50:23.792369 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.791531    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/41fe6d63-9316-4916-b042-87ba7e602167-run-openvswitch\") pod \"ovnkube-node-ml9xf\" (UID: \"41fe6d63-9316-4916-b042-87ba7e602167\") " pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf"
Oct 30 21:50:23.792369 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.791585    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/924394f7-5b3b-4767-a1ff-f1d310ece4ed-device-dir\") pod \"aws-ebs-csi-driver-node-lrc2c\" (UID: \"924394f7-5b3b-4767-a1ff-f1d310ece4ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lrc2c"
Oct 30 21:50:23.792369 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.791612    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5f468f3c-bf3d-437c-8ec8-a5d56f448d99-multus-conf-dir\") pod \"multus-p7zpx\" (UID: \"5f468f3c-bf3d-437c-8ec8-a5d56f448d99\") " pod="openshift-multus/multus-p7zpx"
Oct 30 21:50:23.792369 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.791639    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d947d3db-ab6e-4874-8817-f8c74be138a1-run\") pod \"tuned-22pl2\" (UID: \"d947d3db-ab6e-4874-8817-f8c74be138a1\") " pod="openshift-cluster-node-tuning-operator/tuned-22pl2"
Oct 30 21:50:23.792369 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.791663    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5f468f3c-bf3d-437c-8ec8-a5d56f448d99-multus-conf-dir\") pod \"multus-p7zpx\" (UID: \"5f468f3c-bf3d-437c-8ec8-a5d56f448d99\") " pod="openshift-multus/multus-p7zpx"
Oct 30 21:50:23.792369 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.791664    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5f468f3c-bf3d-437c-8ec8-a5d56f448d99-host-var-lib-cni-bin\") pod \"multus-p7zpx\" (UID: \"5f468f3c-bf3d-437c-8ec8-a5d56f448d99\") " pod="openshift-multus/multus-p7zpx"
Oct 30 21:50:23.792369 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.791671    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/924394f7-5b3b-4767-a1ff-f1d310ece4ed-device-dir\") pod \"aws-ebs-csi-driver-node-lrc2c\" (UID: \"924394f7-5b3b-4767-a1ff-f1d310ece4ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lrc2c"
Oct 30 21:50:23.792369 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.791688    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f938a93b-29dd-4e7d-b8e9-444df68acf03-node-exporter-wtmp\") pod \"node-exporter-7j5j6\" (UID: \"f938a93b-29dd-4e7d-b8e9-444df68acf03\") " pod="openshift-monitoring/node-exporter-7j5j6"
Oct 30 21:50:23.792369 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.791700    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d947d3db-ab6e-4874-8817-f8c74be138a1-run\") pod \"tuned-22pl2\" (UID: \"d947d3db-ab6e-4874-8817-f8c74be138a1\") " pod="openshift-cluster-node-tuning-operator/tuned-22pl2"
Oct 30 21:50:23.792369 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.791702    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5f468f3c-bf3d-437c-8ec8-a5d56f448d99-host-var-lib-cni-bin\") pod \"multus-p7zpx\" (UID: \"5f468f3c-bf3d-437c-8ec8-a5d56f448d99\") " pod="openshift-multus/multus-p7zpx"
Oct 30 21:50:23.792369 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.791715    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5f468f3c-bf3d-437c-8ec8-a5d56f448d99-system-cni-dir\") pod \"multus-p7zpx\" (UID: \"5f468f3c-bf3d-437c-8ec8-a5d56f448d99\") " pod="openshift-multus/multus-p7zpx"
Oct 30 21:50:23.792369 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.791737    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d947d3db-ab6e-4874-8817-f8c74be138a1-etc-modprobe-d\") pod \"tuned-22pl2\" (UID: \"d947d3db-ab6e-4874-8817-f8c74be138a1\") " pod="openshift-cluster-node-tuning-operator/tuned-22pl2"
Oct 30 21:50:23.792369 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.791742    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f938a93b-29dd-4e7d-b8e9-444df68acf03-node-exporter-wtmp\") pod \"node-exporter-7j5j6\" (UID: \"f938a93b-29dd-4e7d-b8e9-444df68acf03\") " pod="openshift-monitoring/node-exporter-7j5j6"
Oct 30 21:50:23.792369 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.791760    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d947d3db-ab6e-4874-8817-f8c74be138a1-etc-sysctl-d\") pod \"tuned-22pl2\" (UID: \"d947d3db-ab6e-4874-8817-f8c74be138a1\") " pod="openshift-cluster-node-tuning-operator/tuned-22pl2"
Oct 30 21:50:23.792369 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.791797    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5f468f3c-bf3d-437c-8ec8-a5d56f448d99-system-cni-dir\") pod \"multus-p7zpx\" (UID: \"5f468f3c-bf3d-437c-8ec8-a5d56f448d99\") " pod="openshift-multus/multus-p7zpx"
Oct 30 21:50:23.792369 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.791835    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d947d3db-ab6e-4874-8817-f8c74be138a1-etc-modprobe-d\") pod \"tuned-22pl2\" (UID: \"d947d3db-ab6e-4874-8817-f8c74be138a1\") " pod="openshift-cluster-node-tuning-operator/tuned-22pl2"
Oct 30 21:50:23.792369 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.791890    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d947d3db-ab6e-4874-8817-f8c74be138a1-etc-sysctl-d\") pod \"tuned-22pl2\" (UID: \"d947d3db-ab6e-4874-8817-f8c74be138a1\") " pod="openshift-cluster-node-tuning-operator/tuned-22pl2"
Oct 30 21:50:23.881962 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:23.880843    5678 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 21:50:25.247250 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:25.247218    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-fdmdz"
Oct 30 21:50:25.291320 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:25.291293    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-69874d4cdd-zhlp5"
Oct 30 21:50:25.316811 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:25.316774    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-69874d4cdd-zhlp5"]
Oct 30 21:50:26.541468 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:26.541445    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf"
Oct 30 21:50:26.603810 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:26.603777    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ef7ae922-6da7-4817-ba91-35e67b6ac8c5","Type":"ContainerStarted","Data":"7f01cffd7d46651522100fea81bb705ec595c21b7479821b5b352982cd7d4988"}
Oct 30 21:50:27.610840 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:27.610809    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ef7ae922-6da7-4817-ba91-35e67b6ac8c5","Type":"ContainerStarted","Data":"9c8be5ed7b18aa2995dfcea5a35d31fa9f057c30716bfa5b89553003d7cc68d2"}
Oct 30 21:50:27.611206 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:27.610895    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ef7ae922-6da7-4817-ba91-35e67b6ac8c5","Type":"ContainerStarted","Data":"1ff12d7276ddc77131dc926b364267852a917f820b7e54191df7b71e90f45ca1"}
Oct 30 21:50:27.611206 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:27.610927    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ef7ae922-6da7-4817-ba91-35e67b6ac8c5","Type":"ContainerStarted","Data":"b6e73fa7c904d0a775556c74faaf3620b284efffebf5489198770681155a28d0"}
Oct 30 21:50:27.833815 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:27.833785    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-dc7c7d47d-ghdn7"
Oct 30 21:50:27.942482 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:27.942450    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7cbdbcf4b5-rmbsz"
Oct 30 21:50:28.614378 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:28.614351    5678 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 21:50:28.629921 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:28.629896    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:28.644869 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:28.644843    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:29.936952 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:29.936925    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf"
Oct 30 21:50:29.937384 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:29.937167    5678 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 21:50:30.265490 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:30.265383    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-w9tx8"
Oct 30 21:50:30.965954 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:30.965922    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-w72c9"
Oct 30 21:50:31.794206 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:31.794174    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7cbdbcf4b5-rmbsz"
Oct 30 21:50:33.056247 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:33.056215    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ml9xf"
Oct 30 21:50:37.947416 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:37.947373    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7cbdbcf4b5-rmbsz"
Oct 30 21:50:37.950975 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:37.950955    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7cbdbcf4b5-rmbsz"
Oct 30 21:50:39.192919 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:39.192881    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-dc7c7d47d-ghdn7"
Oct 30 21:50:40.569285 ip-10-0-128-245 kubenswrapper[5678]: E1030 21:50:40.569246    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 21:50:42.001646 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.001608    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 21:50:42.002624 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.002587    5678 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5138fe8c-9927-4d05-8fe5-5556072ed595" containerName="alertmanager" containerID="cri-o://ebbf6d346fb1ef77b512d07e571efc8794da78e6d86fc712a611858d3530ec5d" gracePeriod=120
Oct 30 21:50:42.003030 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.003000    5678 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5138fe8c-9927-4d05-8fe5-5556072ed595" containerName="kube-rbac-proxy" containerID="cri-o://4226791aeb98d543049bf57efc9d5e8b19e4cf53528ff2a742fda907deb4c396" gracePeriod=120
Oct 30 21:50:42.003225 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.003186    5678 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5138fe8c-9927-4d05-8fe5-5556072ed595" containerName="kube-rbac-proxy-web" containerID="cri-o://a93d15733e8cc09f6a51bfb22088505a1330edba1d3d0d6106aea54679c62b22" gracePeriod=120
Oct 30 21:50:42.003366 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.003327    5678 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5138fe8c-9927-4d05-8fe5-5556072ed595" containerName="config-reloader" containerID="cri-o://a4213395543a6ef339728f8523f33f15faf75bdee21587970d4f87f953cf526e" gracePeriod=120
Oct 30 21:50:42.003458 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.003361    5678 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5138fe8c-9927-4d05-8fe5-5556072ed595" containerName="prom-label-proxy" containerID="cri-o://0026372bec6ebf4cc2f80b456ae15eafc47d5675ef85d041318db9169cdec013" gracePeriod=120
Oct 30 21:50:42.003518 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.003453    5678 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5138fe8c-9927-4d05-8fe5-5556072ed595" containerName="kube-rbac-proxy-metric" containerID="cri-o://d2766685272356bcd2c86bf5bf3b1cce9d467808eba84132b88fbbe31d1643ef" gracePeriod=120
Oct 30 21:50:42.267228 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.267203    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:50:42.358539 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.358516    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5138fe8c-9927-4d05-8fe5-5556072ed595-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5138fe8c-9927-4d05-8fe5-5556072ed595-alertmanager-main-db\") pod \"5138fe8c-9927-4d05-8fe5-5556072ed595\" (UID: \"5138fe8c-9927-4d05-8fe5-5556072ed595\") "
Oct 30 21:50:42.358692 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.358554    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5138fe8c-9927-4d05-8fe5-5556072ed595-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5138fe8c-9927-4d05-8fe5-5556072ed595-cluster-tls-config\") pod \"5138fe8c-9927-4d05-8fe5-5556072ed595\" (UID: \"5138fe8c-9927-4d05-8fe5-5556072ed595\") "
Oct 30 21:50:42.358692 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.358591    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5138fe8c-9927-4d05-8fe5-5556072ed595-kube-api-access-6v9rf\" (UniqueName: \"kubernetes.io/projected/5138fe8c-9927-4d05-8fe5-5556072ed595-kube-api-access-6v9rf\") pod \"5138fe8c-9927-4d05-8fe5-5556072ed595\" (UID: \"5138fe8c-9927-4d05-8fe5-5556072ed595\") "
Oct 30 21:50:42.358692 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.358637    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5138fe8c-9927-4d05-8fe5-5556072ed595-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5138fe8c-9927-4d05-8fe5-5556072ed595-secret-alertmanager-main-tls\") pod \"5138fe8c-9927-4d05-8fe5-5556072ed595\" (UID: \"5138fe8c-9927-4d05-8fe5-5556072ed595\") "
Oct 30 21:50:42.358692 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.358667    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5138fe8c-9927-4d05-8fe5-5556072ed595-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5138fe8c-9927-4d05-8fe5-5556072ed595-secret-alertmanager-kube-rbac-proxy-web\") pod \"5138fe8c-9927-4d05-8fe5-5556072ed595\" (UID: \"5138fe8c-9927-4d05-8fe5-5556072ed595\") "
Oct 30 21:50:42.358902 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.358696    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5138fe8c-9927-4d05-8fe5-5556072ed595-config-volume\" (UniqueName: \"kubernetes.io/secret/5138fe8c-9927-4d05-8fe5-5556072ed595-config-volume\") pod \"5138fe8c-9927-4d05-8fe5-5556072ed595\" (UID: \"5138fe8c-9927-4d05-8fe5-5556072ed595\") "
Oct 30 21:50:42.358902 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.358762    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5138fe8c-9927-4d05-8fe5-5556072ed595-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5138fe8c-9927-4d05-8fe5-5556072ed595-alertmanager-trusted-ca-bundle\") pod \"5138fe8c-9927-4d05-8fe5-5556072ed595\" (UID: \"5138fe8c-9927-4d05-8fe5-5556072ed595\") "
Oct 30 21:50:42.358902 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.358798    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5138fe8c-9927-4d05-8fe5-5556072ed595-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5138fe8c-9927-4d05-8fe5-5556072ed595-metrics-client-ca\") pod \"5138fe8c-9927-4d05-8fe5-5556072ed595\" (UID: \"5138fe8c-9927-4d05-8fe5-5556072ed595\") "
Oct 30 21:50:42.358902 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.358841    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5138fe8c-9927-4d05-8fe5-5556072ed595-web-config\" (UniqueName: \"kubernetes.io/secret/5138fe8c-9927-4d05-8fe5-5556072ed595-web-config\") pod \"5138fe8c-9927-4d05-8fe5-5556072ed595\" (UID: \"5138fe8c-9927-4d05-8fe5-5556072ed595\") "
Oct 30 21:50:42.358902 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.358868    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5138fe8c-9927-4d05-8fe5-5556072ed595-config-out\" (UniqueName: \"kubernetes.io/empty-dir/5138fe8c-9927-4d05-8fe5-5556072ed595-config-out\") pod \"5138fe8c-9927-4d05-8fe5-5556072ed595\" (UID: \"5138fe8c-9927-4d05-8fe5-5556072ed595\") "
Oct 30 21:50:42.359134 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.358907    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5138fe8c-9927-4d05-8fe5-5556072ed595-tls-assets\" (UniqueName: \"kubernetes.io/projected/5138fe8c-9927-4d05-8fe5-5556072ed595-tls-assets\") pod \"5138fe8c-9927-4d05-8fe5-5556072ed595\" (UID: \"5138fe8c-9927-4d05-8fe5-5556072ed595\") "
Oct 30 21:50:42.359134 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.358939    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5138fe8c-9927-4d05-8fe5-5556072ed595-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5138fe8c-9927-4d05-8fe5-5556072ed595-secret-alertmanager-kube-rbac-proxy-metric\") pod \"5138fe8c-9927-4d05-8fe5-5556072ed595\" (UID: \"5138fe8c-9927-4d05-8fe5-5556072ed595\") "
Oct 30 21:50:42.359134 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.358982    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5138fe8c-9927-4d05-8fe5-5556072ed595-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5138fe8c-9927-4d05-8fe5-5556072ed595-secret-alertmanager-kube-rbac-proxy\") pod \"5138fe8c-9927-4d05-8fe5-5556072ed595\" (UID: \"5138fe8c-9927-4d05-8fe5-5556072ed595\") "
Oct 30 21:50:42.359281 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.358835    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5138fe8c-9927-4d05-8fe5-5556072ed595-alertmanager-main-db" pod "5138fe8c-9927-4d05-8fe5-5556072ed595" (UID: "5138fe8c-9927-4d05-8fe5-5556072ed595"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:50:42.359281 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.359189    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5138fe8c-9927-4d05-8fe5-5556072ed595-alertmanager-trusted-ca-bundle" pod "5138fe8c-9927-4d05-8fe5-5556072ed595" (UID: "5138fe8c-9927-4d05-8fe5-5556072ed595"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:50:42.359659 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.359634    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5138fe8c-9927-4d05-8fe5-5556072ed595-metrics-client-ca" pod "5138fe8c-9927-4d05-8fe5-5556072ed595" (UID: "5138fe8c-9927-4d05-8fe5-5556072ed595"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:50:42.362333 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.362288    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5138fe8c-9927-4d05-8fe5-5556072ed595-secret-alertmanager-main-tls" pod "5138fe8c-9927-4d05-8fe5-5556072ed595" (UID: "5138fe8c-9927-4d05-8fe5-5556072ed595"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:50:42.362626 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.362525    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5138fe8c-9927-4d05-8fe5-5556072ed595-kube-api-access-6v9rf" pod "5138fe8c-9927-4d05-8fe5-5556072ed595" (UID: "5138fe8c-9927-4d05-8fe5-5556072ed595"). InnerVolumeSpecName "kube-api-access-6v9rf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:50:42.362730 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.362695    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5138fe8c-9927-4d05-8fe5-5556072ed595-secret-alertmanager-kube-rbac-proxy" pod "5138fe8c-9927-4d05-8fe5-5556072ed595" (UID: "5138fe8c-9927-4d05-8fe5-5556072ed595"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:50:42.362927 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.362905    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5138fe8c-9927-4d05-8fe5-5556072ed595-secret-alertmanager-kube-rbac-proxy-metric" pod "5138fe8c-9927-4d05-8fe5-5556072ed595" (UID: "5138fe8c-9927-4d05-8fe5-5556072ed595"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:50:42.363073 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.363050    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5138fe8c-9927-4d05-8fe5-5556072ed595-config-volume" pod "5138fe8c-9927-4d05-8fe5-5556072ed595" (UID: "5138fe8c-9927-4d05-8fe5-5556072ed595"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:50:42.363210 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.363180    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5138fe8c-9927-4d05-8fe5-5556072ed595-secret-alertmanager-kube-rbac-proxy-web" pod "5138fe8c-9927-4d05-8fe5-5556072ed595" (UID: "5138fe8c-9927-4d05-8fe5-5556072ed595"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:50:42.363304 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.363216    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5138fe8c-9927-4d05-8fe5-5556072ed595-config-out" pod "5138fe8c-9927-4d05-8fe5-5556072ed595" (UID: "5138fe8c-9927-4d05-8fe5-5556072ed595"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:50:42.363304 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.363245    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5138fe8c-9927-4d05-8fe5-5556072ed595-tls-assets" pod "5138fe8c-9927-4d05-8fe5-5556072ed595" (UID: "5138fe8c-9927-4d05-8fe5-5556072ed595"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:50:42.366145 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.366037    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5138fe8c-9927-4d05-8fe5-5556072ed595-cluster-tls-config" pod "5138fe8c-9927-4d05-8fe5-5556072ed595" (UID: "5138fe8c-9927-4d05-8fe5-5556072ed595"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:50:42.372816 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.372794    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5138fe8c-9927-4d05-8fe5-5556072ed595-web-config" pod "5138fe8c-9927-4d05-8fe5-5556072ed595" (UID: "5138fe8c-9927-4d05-8fe5-5556072ed595"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:50:42.460178 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.460145    5678 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5138fe8c-9927-4d05-8fe5-5556072ed595-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:50:42.460178 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.460171    5678 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5138fe8c-9927-4d05-8fe5-5556072ed595-alertmanager-main-db\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:50:42.460178 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.460182    5678 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5138fe8c-9927-4d05-8fe5-5556072ed595-cluster-tls-config\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:50:42.460384 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.460194    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6v9rf\" (UniqueName: \"kubernetes.io/projected/5138fe8c-9927-4d05-8fe5-5556072ed595-kube-api-access-6v9rf\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:50:42.460384 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.460208    5678 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5138fe8c-9927-4d05-8fe5-5556072ed595-secret-alertmanager-main-tls\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:50:42.460384 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.460222    5678 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5138fe8c-9927-4d05-8fe5-5556072ed595-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:50:42.460384 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.460237    5678 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5138fe8c-9927-4d05-8fe5-5556072ed595-config-volume\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:50:42.460384 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.460254    5678 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5138fe8c-9927-4d05-8fe5-5556072ed595-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:50:42.460384 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.460269    5678 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5138fe8c-9927-4d05-8fe5-5556072ed595-metrics-client-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:50:42.460384 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.460312    5678 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5138fe8c-9927-4d05-8fe5-5556072ed595-web-config\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:50:42.460384 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.460327    5678 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5138fe8c-9927-4d05-8fe5-5556072ed595-config-out\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:50:42.460384 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.460339    5678 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5138fe8c-9927-4d05-8fe5-5556072ed595-tls-assets\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:50:42.460384 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.460348    5678 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5138fe8c-9927-4d05-8fe5-5556072ed595-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:50:42.667757 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.667725    5678 generic.go:356] "Generic (PLEG): container finished" podID="5138fe8c-9927-4d05-8fe5-5556072ed595" containerID="0026372bec6ebf4cc2f80b456ae15eafc47d5675ef85d041318db9169cdec013" exitCode=0
Oct 30 21:50:42.667757 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.667748    5678 generic.go:356] "Generic (PLEG): container finished" podID="5138fe8c-9927-4d05-8fe5-5556072ed595" containerID="d2766685272356bcd2c86bf5bf3b1cce9d467808eba84132b88fbbe31d1643ef" exitCode=0
Oct 30 21:50:42.667757 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.667756    5678 generic.go:356] "Generic (PLEG): container finished" podID="5138fe8c-9927-4d05-8fe5-5556072ed595" containerID="4226791aeb98d543049bf57efc9d5e8b19e4cf53528ff2a742fda907deb4c396" exitCode=0
Oct 30 21:50:42.667757 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.667762    5678 generic.go:356] "Generic (PLEG): container finished" podID="5138fe8c-9927-4d05-8fe5-5556072ed595" containerID="a93d15733e8cc09f6a51bfb22088505a1330edba1d3d0d6106aea54679c62b22" exitCode=0
Oct 30 21:50:42.668123 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.667773    5678 generic.go:356] "Generic (PLEG): container finished" podID="5138fe8c-9927-4d05-8fe5-5556072ed595" containerID="a4213395543a6ef339728f8523f33f15faf75bdee21587970d4f87f953cf526e" exitCode=0
Oct 30 21:50:42.668123 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.667781    5678 generic.go:356] "Generic (PLEG): container finished" podID="5138fe8c-9927-4d05-8fe5-5556072ed595" containerID="ebbf6d346fb1ef77b512d07e571efc8794da78e6d86fc712a611858d3530ec5d" exitCode=0
Oct 30 21:50:42.668123 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.667793    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:50:42.668123 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.667832    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5138fe8c-9927-4d05-8fe5-5556072ed595","Type":"ContainerDied","Data":"0026372bec6ebf4cc2f80b456ae15eafc47d5675ef85d041318db9169cdec013"}
Oct 30 21:50:42.668123 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.667893    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5138fe8c-9927-4d05-8fe5-5556072ed595","Type":"ContainerDied","Data":"d2766685272356bcd2c86bf5bf3b1cce9d467808eba84132b88fbbe31d1643ef"}
Oct 30 21:50:42.668123 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.667919    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5138fe8c-9927-4d05-8fe5-5556072ed595","Type":"ContainerDied","Data":"4226791aeb98d543049bf57efc9d5e8b19e4cf53528ff2a742fda907deb4c396"}
Oct 30 21:50:42.668123 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.667947    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5138fe8c-9927-4d05-8fe5-5556072ed595","Type":"ContainerDied","Data":"a93d15733e8cc09f6a51bfb22088505a1330edba1d3d0d6106aea54679c62b22"}
Oct 30 21:50:42.668123 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.667957    5678 scope.go:122] "RemoveContainer" containerID="0026372bec6ebf4cc2f80b456ae15eafc47d5675ef85d041318db9169cdec013"
Oct 30 21:50:42.668123 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.667991    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5138fe8c-9927-4d05-8fe5-5556072ed595","Type":"ContainerDied","Data":"a4213395543a6ef339728f8523f33f15faf75bdee21587970d4f87f953cf526e"}
Oct 30 21:50:42.668123 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.668032    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5138fe8c-9927-4d05-8fe5-5556072ed595","Type":"ContainerDied","Data":"ebbf6d346fb1ef77b512d07e571efc8794da78e6d86fc712a611858d3530ec5d"}
Oct 30 21:50:42.668123 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.668071    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5138fe8c-9927-4d05-8fe5-5556072ed595","Type":"ContainerDied","Data":"dc24a46b097bebb5444228144f254835785f13e6c1da310ba365dff351a254f6"}
Oct 30 21:50:42.675568 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.675552    5678 scope.go:122] "RemoveContainer" containerID="d2766685272356bcd2c86bf5bf3b1cce9d467808eba84132b88fbbe31d1643ef"
Oct 30 21:50:42.682318 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.682303    5678 scope.go:122] "RemoveContainer" containerID="4226791aeb98d543049bf57efc9d5e8b19e4cf53528ff2a742fda907deb4c396"
Oct 30 21:50:42.688730 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.688716    5678 scope.go:122] "RemoveContainer" containerID="a93d15733e8cc09f6a51bfb22088505a1330edba1d3d0d6106aea54679c62b22"
Oct 30 21:50:42.695115 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.695098    5678 scope.go:122] "RemoveContainer" containerID="a4213395543a6ef339728f8523f33f15faf75bdee21587970d4f87f953cf526e"
Oct 30 21:50:42.701348 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.701327    5678 scope.go:122] "RemoveContainer" containerID="ebbf6d346fb1ef77b512d07e571efc8794da78e6d86fc712a611858d3530ec5d"
Oct 30 21:50:42.708202 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.708183    5678 scope.go:122] "RemoveContainer" containerID="f7f271fecf3370b4e1a7117367120bc8e1a3ba6129d0a2171e8c002bee61190a"
Oct 30 21:50:42.714544 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.714529    5678 scope.go:122] "RemoveContainer" containerID="0026372bec6ebf4cc2f80b456ae15eafc47d5675ef85d041318db9169cdec013"
Oct 30 21:50:42.714791 ip-10-0-128-245 kubenswrapper[5678]: E1030 21:50:42.714774    5678 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0026372bec6ebf4cc2f80b456ae15eafc47d5675ef85d041318db9169cdec013\": container with ID starting with 0026372bec6ebf4cc2f80b456ae15eafc47d5675ef85d041318db9169cdec013 not found: ID does not exist" containerID="0026372bec6ebf4cc2f80b456ae15eafc47d5675ef85d041318db9169cdec013"
Oct 30 21:50:42.714834 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.714800    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0026372bec6ebf4cc2f80b456ae15eafc47d5675ef85d041318db9169cdec013"} err="failed to get container status \"0026372bec6ebf4cc2f80b456ae15eafc47d5675ef85d041318db9169cdec013\": rpc error: code = NotFound desc = could not find container \"0026372bec6ebf4cc2f80b456ae15eafc47d5675ef85d041318db9169cdec013\": container with ID starting with 0026372bec6ebf4cc2f80b456ae15eafc47d5675ef85d041318db9169cdec013 not found: ID does not exist"
Oct 30 21:50:42.714834 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.714831    5678 scope.go:122] "RemoveContainer" containerID="d2766685272356bcd2c86bf5bf3b1cce9d467808eba84132b88fbbe31d1643ef"
Oct 30 21:50:42.715035 ip-10-0-128-245 kubenswrapper[5678]: E1030 21:50:42.715020    5678 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2766685272356bcd2c86bf5bf3b1cce9d467808eba84132b88fbbe31d1643ef\": container with ID starting with d2766685272356bcd2c86bf5bf3b1cce9d467808eba84132b88fbbe31d1643ef not found: ID does not exist" containerID="d2766685272356bcd2c86bf5bf3b1cce9d467808eba84132b88fbbe31d1643ef"
Oct 30 21:50:42.715077 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.715040    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2766685272356bcd2c86bf5bf3b1cce9d467808eba84132b88fbbe31d1643ef"} err="failed to get container status \"d2766685272356bcd2c86bf5bf3b1cce9d467808eba84132b88fbbe31d1643ef\": rpc error: code = NotFound desc = could not find container \"d2766685272356bcd2c86bf5bf3b1cce9d467808eba84132b88fbbe31d1643ef\": container with ID starting with d2766685272356bcd2c86bf5bf3b1cce9d467808eba84132b88fbbe31d1643ef not found: ID does not exist"
Oct 30 21:50:42.715077 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.715052    5678 scope.go:122] "RemoveContainer" containerID="4226791aeb98d543049bf57efc9d5e8b19e4cf53528ff2a742fda907deb4c396"
Oct 30 21:50:42.715270 ip-10-0-128-245 kubenswrapper[5678]: E1030 21:50:42.715254    5678 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4226791aeb98d543049bf57efc9d5e8b19e4cf53528ff2a742fda907deb4c396\": container with ID starting with 4226791aeb98d543049bf57efc9d5e8b19e4cf53528ff2a742fda907deb4c396 not found: ID does not exist" containerID="4226791aeb98d543049bf57efc9d5e8b19e4cf53528ff2a742fda907deb4c396"
Oct 30 21:50:42.715311 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.715274    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4226791aeb98d543049bf57efc9d5e8b19e4cf53528ff2a742fda907deb4c396"} err="failed to get container status \"4226791aeb98d543049bf57efc9d5e8b19e4cf53528ff2a742fda907deb4c396\": rpc error: code = NotFound desc = could not find container \"4226791aeb98d543049bf57efc9d5e8b19e4cf53528ff2a742fda907deb4c396\": container with ID starting with 4226791aeb98d543049bf57efc9d5e8b19e4cf53528ff2a742fda907deb4c396 not found: ID does not exist"
Oct 30 21:50:42.715311 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.715284    5678 scope.go:122] "RemoveContainer" containerID="a93d15733e8cc09f6a51bfb22088505a1330edba1d3d0d6106aea54679c62b22"
Oct 30 21:50:42.715515 ip-10-0-128-245 kubenswrapper[5678]: E1030 21:50:42.715496    5678 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a93d15733e8cc09f6a51bfb22088505a1330edba1d3d0d6106aea54679c62b22\": container with ID starting with a93d15733e8cc09f6a51bfb22088505a1330edba1d3d0d6106aea54679c62b22 not found: ID does not exist" containerID="a93d15733e8cc09f6a51bfb22088505a1330edba1d3d0d6106aea54679c62b22"
Oct 30 21:50:42.715562 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.715520    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a93d15733e8cc09f6a51bfb22088505a1330edba1d3d0d6106aea54679c62b22"} err="failed to get container status \"a93d15733e8cc09f6a51bfb22088505a1330edba1d3d0d6106aea54679c62b22\": rpc error: code = NotFound desc = could not find container \"a93d15733e8cc09f6a51bfb22088505a1330edba1d3d0d6106aea54679c62b22\": container with ID starting with a93d15733e8cc09f6a51bfb22088505a1330edba1d3d0d6106aea54679c62b22 not found: ID does not exist"
Oct 30 21:50:42.715562 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.715535    5678 scope.go:122] "RemoveContainer" containerID="a4213395543a6ef339728f8523f33f15faf75bdee21587970d4f87f953cf526e"
Oct 30 21:50:42.715752 ip-10-0-128-245 kubenswrapper[5678]: E1030 21:50:42.715737    5678 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4213395543a6ef339728f8523f33f15faf75bdee21587970d4f87f953cf526e\": container with ID starting with a4213395543a6ef339728f8523f33f15faf75bdee21587970d4f87f953cf526e not found: ID does not exist" containerID="a4213395543a6ef339728f8523f33f15faf75bdee21587970d4f87f953cf526e"
Oct 30 21:50:42.715788 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.715757    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4213395543a6ef339728f8523f33f15faf75bdee21587970d4f87f953cf526e"} err="failed to get container status \"a4213395543a6ef339728f8523f33f15faf75bdee21587970d4f87f953cf526e\": rpc error: code = NotFound desc = could not find container \"a4213395543a6ef339728f8523f33f15faf75bdee21587970d4f87f953cf526e\": container with ID starting with a4213395543a6ef339728f8523f33f15faf75bdee21587970d4f87f953cf526e not found: ID does not exist"
Oct 30 21:50:42.715788 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.715769    5678 scope.go:122] "RemoveContainer" containerID="ebbf6d346fb1ef77b512d07e571efc8794da78e6d86fc712a611858d3530ec5d"
Oct 30 21:50:42.715991 ip-10-0-128-245 kubenswrapper[5678]: E1030 21:50:42.715969    5678 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebbf6d346fb1ef77b512d07e571efc8794da78e6d86fc712a611858d3530ec5d\": container with ID starting with ebbf6d346fb1ef77b512d07e571efc8794da78e6d86fc712a611858d3530ec5d not found: ID does not exist" containerID="ebbf6d346fb1ef77b512d07e571efc8794da78e6d86fc712a611858d3530ec5d"
Oct 30 21:50:42.716030 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.715996    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebbf6d346fb1ef77b512d07e571efc8794da78e6d86fc712a611858d3530ec5d"} err="failed to get container status \"ebbf6d346fb1ef77b512d07e571efc8794da78e6d86fc712a611858d3530ec5d\": rpc error: code = NotFound desc = could not find container \"ebbf6d346fb1ef77b512d07e571efc8794da78e6d86fc712a611858d3530ec5d\": container with ID starting with ebbf6d346fb1ef77b512d07e571efc8794da78e6d86fc712a611858d3530ec5d not found: ID does not exist"
Oct 30 21:50:42.716030 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.716007    5678 scope.go:122] "RemoveContainer" containerID="f7f271fecf3370b4e1a7117367120bc8e1a3ba6129d0a2171e8c002bee61190a"
Oct 30 21:50:42.716198 ip-10-0-128-245 kubenswrapper[5678]: E1030 21:50:42.716184    5678 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7f271fecf3370b4e1a7117367120bc8e1a3ba6129d0a2171e8c002bee61190a\": container with ID starting with f7f271fecf3370b4e1a7117367120bc8e1a3ba6129d0a2171e8c002bee61190a not found: ID does not exist" containerID="f7f271fecf3370b4e1a7117367120bc8e1a3ba6129d0a2171e8c002bee61190a"
Oct 30 21:50:42.716241 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.716201    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7f271fecf3370b4e1a7117367120bc8e1a3ba6129d0a2171e8c002bee61190a"} err="failed to get container status \"f7f271fecf3370b4e1a7117367120bc8e1a3ba6129d0a2171e8c002bee61190a\": rpc error: code = NotFound desc = could not find container \"f7f271fecf3370b4e1a7117367120bc8e1a3ba6129d0a2171e8c002bee61190a\": container with ID starting with f7f271fecf3370b4e1a7117367120bc8e1a3ba6129d0a2171e8c002bee61190a not found: ID does not exist"
Oct 30 21:50:42.716241 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.716211    5678 scope.go:122] "RemoveContainer" containerID="0026372bec6ebf4cc2f80b456ae15eafc47d5675ef85d041318db9169cdec013"
Oct 30 21:50:42.716394 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.716378    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0026372bec6ebf4cc2f80b456ae15eafc47d5675ef85d041318db9169cdec013"} err="failed to get container status \"0026372bec6ebf4cc2f80b456ae15eafc47d5675ef85d041318db9169cdec013\": rpc error: code = NotFound desc = could not find container \"0026372bec6ebf4cc2f80b456ae15eafc47d5675ef85d041318db9169cdec013\": container with ID starting with 0026372bec6ebf4cc2f80b456ae15eafc47d5675ef85d041318db9169cdec013 not found: ID does not exist"
Oct 30 21:50:42.716394 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.716394    5678 scope.go:122] "RemoveContainer" containerID="d2766685272356bcd2c86bf5bf3b1cce9d467808eba84132b88fbbe31d1643ef"
Oct 30 21:50:42.716597 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.716581    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2766685272356bcd2c86bf5bf3b1cce9d467808eba84132b88fbbe31d1643ef"} err="failed to get container status \"d2766685272356bcd2c86bf5bf3b1cce9d467808eba84132b88fbbe31d1643ef\": rpc error: code = NotFound desc = could not find container \"d2766685272356bcd2c86bf5bf3b1cce9d467808eba84132b88fbbe31d1643ef\": container with ID starting with d2766685272356bcd2c86bf5bf3b1cce9d467808eba84132b88fbbe31d1643ef not found: ID does not exist"
Oct 30 21:50:42.716639 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.716597    5678 scope.go:122] "RemoveContainer" containerID="4226791aeb98d543049bf57efc9d5e8b19e4cf53528ff2a742fda907deb4c396"
Oct 30 21:50:42.716763 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.716748    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4226791aeb98d543049bf57efc9d5e8b19e4cf53528ff2a742fda907deb4c396"} err="failed to get container status \"4226791aeb98d543049bf57efc9d5e8b19e4cf53528ff2a742fda907deb4c396\": rpc error: code = NotFound desc = could not find container \"4226791aeb98d543049bf57efc9d5e8b19e4cf53528ff2a742fda907deb4c396\": container with ID starting with 4226791aeb98d543049bf57efc9d5e8b19e4cf53528ff2a742fda907deb4c396 not found: ID does not exist"
Oct 30 21:50:42.716817 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.716763    5678 scope.go:122] "RemoveContainer" containerID="a93d15733e8cc09f6a51bfb22088505a1330edba1d3d0d6106aea54679c62b22"
Oct 30 21:50:42.716979 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.716961    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a93d15733e8cc09f6a51bfb22088505a1330edba1d3d0d6106aea54679c62b22"} err="failed to get container status \"a93d15733e8cc09f6a51bfb22088505a1330edba1d3d0d6106aea54679c62b22\": rpc error: code = NotFound desc = could not find container \"a93d15733e8cc09f6a51bfb22088505a1330edba1d3d0d6106aea54679c62b22\": container with ID starting with a93d15733e8cc09f6a51bfb22088505a1330edba1d3d0d6106aea54679c62b22 not found: ID does not exist"
Oct 30 21:50:42.717023 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.716981    5678 scope.go:122] "RemoveContainer" containerID="a4213395543a6ef339728f8523f33f15faf75bdee21587970d4f87f953cf526e"
Oct 30 21:50:42.717192 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.717173    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4213395543a6ef339728f8523f33f15faf75bdee21587970d4f87f953cf526e"} err="failed to get container status \"a4213395543a6ef339728f8523f33f15faf75bdee21587970d4f87f953cf526e\": rpc error: code = NotFound desc = could not find container \"a4213395543a6ef339728f8523f33f15faf75bdee21587970d4f87f953cf526e\": container with ID starting with a4213395543a6ef339728f8523f33f15faf75bdee21587970d4f87f953cf526e not found: ID does not exist"
Oct 30 21:50:42.717234 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.717194    5678 scope.go:122] "RemoveContainer" containerID="ebbf6d346fb1ef77b512d07e571efc8794da78e6d86fc712a611858d3530ec5d"
Oct 30 21:50:42.717462 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.717379    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebbf6d346fb1ef77b512d07e571efc8794da78e6d86fc712a611858d3530ec5d"} err="failed to get container status \"ebbf6d346fb1ef77b512d07e571efc8794da78e6d86fc712a611858d3530ec5d\": rpc error: code = NotFound desc = could not find container \"ebbf6d346fb1ef77b512d07e571efc8794da78e6d86fc712a611858d3530ec5d\": container with ID starting with ebbf6d346fb1ef77b512d07e571efc8794da78e6d86fc712a611858d3530ec5d not found: ID does not exist"
Oct 30 21:50:42.717462 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.717426    5678 scope.go:122] "RemoveContainer" containerID="f7f271fecf3370b4e1a7117367120bc8e1a3ba6129d0a2171e8c002bee61190a"
Oct 30 21:50:42.717634 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.717618    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7f271fecf3370b4e1a7117367120bc8e1a3ba6129d0a2171e8c002bee61190a"} err="failed to get container status \"f7f271fecf3370b4e1a7117367120bc8e1a3ba6129d0a2171e8c002bee61190a\": rpc error: code = NotFound desc = could not find container \"f7f271fecf3370b4e1a7117367120bc8e1a3ba6129d0a2171e8c002bee61190a\": container with ID starting with f7f271fecf3370b4e1a7117367120bc8e1a3ba6129d0a2171e8c002bee61190a not found: ID does not exist"
Oct 30 21:50:42.717671 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.717634    5678 scope.go:122] "RemoveContainer" containerID="0026372bec6ebf4cc2f80b456ae15eafc47d5675ef85d041318db9169cdec013"
Oct 30 21:50:42.717878 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.717858    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0026372bec6ebf4cc2f80b456ae15eafc47d5675ef85d041318db9169cdec013"} err="failed to get container status \"0026372bec6ebf4cc2f80b456ae15eafc47d5675ef85d041318db9169cdec013\": rpc error: code = NotFound desc = could not find container \"0026372bec6ebf4cc2f80b456ae15eafc47d5675ef85d041318db9169cdec013\": container with ID starting with 0026372bec6ebf4cc2f80b456ae15eafc47d5675ef85d041318db9169cdec013 not found: ID does not exist"
Oct 30 21:50:42.717927 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.717878    5678 scope.go:122] "RemoveContainer" containerID="d2766685272356bcd2c86bf5bf3b1cce9d467808eba84132b88fbbe31d1643ef"
Oct 30 21:50:42.718114 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.718097    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2766685272356bcd2c86bf5bf3b1cce9d467808eba84132b88fbbe31d1643ef"} err="failed to get container status \"d2766685272356bcd2c86bf5bf3b1cce9d467808eba84132b88fbbe31d1643ef\": rpc error: code = NotFound desc = could not find container \"d2766685272356bcd2c86bf5bf3b1cce9d467808eba84132b88fbbe31d1643ef\": container with ID starting with d2766685272356bcd2c86bf5bf3b1cce9d467808eba84132b88fbbe31d1643ef not found: ID does not exist"
Oct 30 21:50:42.718155 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.718114    5678 scope.go:122] "RemoveContainer" containerID="4226791aeb98d543049bf57efc9d5e8b19e4cf53528ff2a742fda907deb4c396"
Oct 30 21:50:42.718326 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.718310    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4226791aeb98d543049bf57efc9d5e8b19e4cf53528ff2a742fda907deb4c396"} err="failed to get container status \"4226791aeb98d543049bf57efc9d5e8b19e4cf53528ff2a742fda907deb4c396\": rpc error: code = NotFound desc = could not find container \"4226791aeb98d543049bf57efc9d5e8b19e4cf53528ff2a742fda907deb4c396\": container with ID starting with 4226791aeb98d543049bf57efc9d5e8b19e4cf53528ff2a742fda907deb4c396 not found: ID does not exist"
Oct 30 21:50:42.718326 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.718325    5678 scope.go:122] "RemoveContainer" containerID="a93d15733e8cc09f6a51bfb22088505a1330edba1d3d0d6106aea54679c62b22"
Oct 30 21:50:42.718598 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.718576    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a93d15733e8cc09f6a51bfb22088505a1330edba1d3d0d6106aea54679c62b22"} err="failed to get container status \"a93d15733e8cc09f6a51bfb22088505a1330edba1d3d0d6106aea54679c62b22\": rpc error: code = NotFound desc = could not find container \"a93d15733e8cc09f6a51bfb22088505a1330edba1d3d0d6106aea54679c62b22\": container with ID starting with a93d15733e8cc09f6a51bfb22088505a1330edba1d3d0d6106aea54679c62b22 not found: ID does not exist"
Oct 30 21:50:42.718598 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.718597    5678 scope.go:122] "RemoveContainer" containerID="a4213395543a6ef339728f8523f33f15faf75bdee21587970d4f87f953cf526e"
Oct 30 21:50:42.718840 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.718826    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4213395543a6ef339728f8523f33f15faf75bdee21587970d4f87f953cf526e"} err="failed to get container status \"a4213395543a6ef339728f8523f33f15faf75bdee21587970d4f87f953cf526e\": rpc error: code = NotFound desc = could not find container \"a4213395543a6ef339728f8523f33f15faf75bdee21587970d4f87f953cf526e\": container with ID starting with a4213395543a6ef339728f8523f33f15faf75bdee21587970d4f87f953cf526e not found: ID does not exist"
Oct 30 21:50:42.718879 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.718840    5678 scope.go:122] "RemoveContainer" containerID="ebbf6d346fb1ef77b512d07e571efc8794da78e6d86fc712a611858d3530ec5d"
Oct 30 21:50:42.719075 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.719057    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebbf6d346fb1ef77b512d07e571efc8794da78e6d86fc712a611858d3530ec5d"} err="failed to get container status \"ebbf6d346fb1ef77b512d07e571efc8794da78e6d86fc712a611858d3530ec5d\": rpc error: code = NotFound desc = could not find container \"ebbf6d346fb1ef77b512d07e571efc8794da78e6d86fc712a611858d3530ec5d\": container with ID starting with ebbf6d346fb1ef77b512d07e571efc8794da78e6d86fc712a611858d3530ec5d not found: ID does not exist"
Oct 30 21:50:42.719116 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.719077    5678 scope.go:122] "RemoveContainer" containerID="f7f271fecf3370b4e1a7117367120bc8e1a3ba6129d0a2171e8c002bee61190a"
Oct 30 21:50:42.719281 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.719261    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7f271fecf3370b4e1a7117367120bc8e1a3ba6129d0a2171e8c002bee61190a"} err="failed to get container status \"f7f271fecf3370b4e1a7117367120bc8e1a3ba6129d0a2171e8c002bee61190a\": rpc error: code = NotFound desc = could not find container \"f7f271fecf3370b4e1a7117367120bc8e1a3ba6129d0a2171e8c002bee61190a\": container with ID starting with f7f271fecf3370b4e1a7117367120bc8e1a3ba6129d0a2171e8c002bee61190a not found: ID does not exist"
Oct 30 21:50:42.719323 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.719282    5678 scope.go:122] "RemoveContainer" containerID="0026372bec6ebf4cc2f80b456ae15eafc47d5675ef85d041318db9169cdec013"
Oct 30 21:50:42.719487 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.719470    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0026372bec6ebf4cc2f80b456ae15eafc47d5675ef85d041318db9169cdec013"} err="failed to get container status \"0026372bec6ebf4cc2f80b456ae15eafc47d5675ef85d041318db9169cdec013\": rpc error: code = NotFound desc = could not find container \"0026372bec6ebf4cc2f80b456ae15eafc47d5675ef85d041318db9169cdec013\": container with ID starting with 0026372bec6ebf4cc2f80b456ae15eafc47d5675ef85d041318db9169cdec013 not found: ID does not exist"
Oct 30 21:50:42.719537 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.719487    5678 scope.go:122] "RemoveContainer" containerID="d2766685272356bcd2c86bf5bf3b1cce9d467808eba84132b88fbbe31d1643ef"
Oct 30 21:50:42.719699 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.719682    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2766685272356bcd2c86bf5bf3b1cce9d467808eba84132b88fbbe31d1643ef"} err="failed to get container status \"d2766685272356bcd2c86bf5bf3b1cce9d467808eba84132b88fbbe31d1643ef\": rpc error: code = NotFound desc = could not find container \"d2766685272356bcd2c86bf5bf3b1cce9d467808eba84132b88fbbe31d1643ef\": container with ID starting with d2766685272356bcd2c86bf5bf3b1cce9d467808eba84132b88fbbe31d1643ef not found: ID does not exist"
Oct 30 21:50:42.719699 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.719699    5678 scope.go:122] "RemoveContainer" containerID="4226791aeb98d543049bf57efc9d5e8b19e4cf53528ff2a742fda907deb4c396"
Oct 30 21:50:42.719910 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.719895    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4226791aeb98d543049bf57efc9d5e8b19e4cf53528ff2a742fda907deb4c396"} err="failed to get container status \"4226791aeb98d543049bf57efc9d5e8b19e4cf53528ff2a742fda907deb4c396\": rpc error: code = NotFound desc = could not find container \"4226791aeb98d543049bf57efc9d5e8b19e4cf53528ff2a742fda907deb4c396\": container with ID starting with 4226791aeb98d543049bf57efc9d5e8b19e4cf53528ff2a742fda907deb4c396 not found: ID does not exist"
Oct 30 21:50:42.719910 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.719909    5678 scope.go:122] "RemoveContainer" containerID="a93d15733e8cc09f6a51bfb22088505a1330edba1d3d0d6106aea54679c62b22"
Oct 30 21:50:42.720119 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.720102    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a93d15733e8cc09f6a51bfb22088505a1330edba1d3d0d6106aea54679c62b22"} err="failed to get container status \"a93d15733e8cc09f6a51bfb22088505a1330edba1d3d0d6106aea54679c62b22\": rpc error: code = NotFound desc = could not find container \"a93d15733e8cc09f6a51bfb22088505a1330edba1d3d0d6106aea54679c62b22\": container with ID starting with a93d15733e8cc09f6a51bfb22088505a1330edba1d3d0d6106aea54679c62b22 not found: ID does not exist"
Oct 30 21:50:42.720119 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.720118    5678 scope.go:122] "RemoveContainer" containerID="a4213395543a6ef339728f8523f33f15faf75bdee21587970d4f87f953cf526e"
Oct 30 21:50:42.720341 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.720322    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4213395543a6ef339728f8523f33f15faf75bdee21587970d4f87f953cf526e"} err="failed to get container status \"a4213395543a6ef339728f8523f33f15faf75bdee21587970d4f87f953cf526e\": rpc error: code = NotFound desc = could not find container \"a4213395543a6ef339728f8523f33f15faf75bdee21587970d4f87f953cf526e\": container with ID starting with a4213395543a6ef339728f8523f33f15faf75bdee21587970d4f87f953cf526e not found: ID does not exist"
Oct 30 21:50:42.720382 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.720342    5678 scope.go:122] "RemoveContainer" containerID="ebbf6d346fb1ef77b512d07e571efc8794da78e6d86fc712a611858d3530ec5d"
Oct 30 21:50:42.720553 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.720538    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebbf6d346fb1ef77b512d07e571efc8794da78e6d86fc712a611858d3530ec5d"} err="failed to get container status \"ebbf6d346fb1ef77b512d07e571efc8794da78e6d86fc712a611858d3530ec5d\": rpc error: code = NotFound desc = could not find container \"ebbf6d346fb1ef77b512d07e571efc8794da78e6d86fc712a611858d3530ec5d\": container with ID starting with ebbf6d346fb1ef77b512d07e571efc8794da78e6d86fc712a611858d3530ec5d not found: ID does not exist"
Oct 30 21:50:42.720595 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.720553    5678 scope.go:122] "RemoveContainer" containerID="f7f271fecf3370b4e1a7117367120bc8e1a3ba6129d0a2171e8c002bee61190a"
Oct 30 21:50:42.720743 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.720729    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7f271fecf3370b4e1a7117367120bc8e1a3ba6129d0a2171e8c002bee61190a"} err="failed to get container status \"f7f271fecf3370b4e1a7117367120bc8e1a3ba6129d0a2171e8c002bee61190a\": rpc error: code = NotFound desc = could not find container \"f7f271fecf3370b4e1a7117367120bc8e1a3ba6129d0a2171e8c002bee61190a\": container with ID starting with f7f271fecf3370b4e1a7117367120bc8e1a3ba6129d0a2171e8c002bee61190a not found: ID does not exist"
Oct 30 21:50:42.720780 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.720743    5678 scope.go:122] "RemoveContainer" containerID="0026372bec6ebf4cc2f80b456ae15eafc47d5675ef85d041318db9169cdec013"
Oct 30 21:50:42.720927 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.720913    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0026372bec6ebf4cc2f80b456ae15eafc47d5675ef85d041318db9169cdec013"} err="failed to get container status \"0026372bec6ebf4cc2f80b456ae15eafc47d5675ef85d041318db9169cdec013\": rpc error: code = NotFound desc = could not find container \"0026372bec6ebf4cc2f80b456ae15eafc47d5675ef85d041318db9169cdec013\": container with ID starting with 0026372bec6ebf4cc2f80b456ae15eafc47d5675ef85d041318db9169cdec013 not found: ID does not exist"
Oct 30 21:50:42.720967 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.720927    5678 scope.go:122] "RemoveContainer" containerID="d2766685272356bcd2c86bf5bf3b1cce9d467808eba84132b88fbbe31d1643ef"
Oct 30 21:50:42.721108 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.721090    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2766685272356bcd2c86bf5bf3b1cce9d467808eba84132b88fbbe31d1643ef"} err="failed to get container status \"d2766685272356bcd2c86bf5bf3b1cce9d467808eba84132b88fbbe31d1643ef\": rpc error: code = NotFound desc = could not find container \"d2766685272356bcd2c86bf5bf3b1cce9d467808eba84132b88fbbe31d1643ef\": container with ID starting with d2766685272356bcd2c86bf5bf3b1cce9d467808eba84132b88fbbe31d1643ef not found: ID does not exist"
Oct 30 21:50:42.721145 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.721108    5678 scope.go:122] "RemoveContainer" containerID="4226791aeb98d543049bf57efc9d5e8b19e4cf53528ff2a742fda907deb4c396"
Oct 30 21:50:42.721298 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.721283    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4226791aeb98d543049bf57efc9d5e8b19e4cf53528ff2a742fda907deb4c396"} err="failed to get container status \"4226791aeb98d543049bf57efc9d5e8b19e4cf53528ff2a742fda907deb4c396\": rpc error: code = NotFound desc = could not find container \"4226791aeb98d543049bf57efc9d5e8b19e4cf53528ff2a742fda907deb4c396\": container with ID starting with 4226791aeb98d543049bf57efc9d5e8b19e4cf53528ff2a742fda907deb4c396 not found: ID does not exist"
Oct 30 21:50:42.721339 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.721298    5678 scope.go:122] "RemoveContainer" containerID="a93d15733e8cc09f6a51bfb22088505a1330edba1d3d0d6106aea54679c62b22"
Oct 30 21:50:42.721490 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.721474    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a93d15733e8cc09f6a51bfb22088505a1330edba1d3d0d6106aea54679c62b22"} err="failed to get container status \"a93d15733e8cc09f6a51bfb22088505a1330edba1d3d0d6106aea54679c62b22\": rpc error: code = NotFound desc = could not find container \"a93d15733e8cc09f6a51bfb22088505a1330edba1d3d0d6106aea54679c62b22\": container with ID starting with a93d15733e8cc09f6a51bfb22088505a1330edba1d3d0d6106aea54679c62b22 not found: ID does not exist"
Oct 30 21:50:42.721536 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.721489    5678 scope.go:122] "RemoveContainer" containerID="a4213395543a6ef339728f8523f33f15faf75bdee21587970d4f87f953cf526e"
Oct 30 21:50:42.721652 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.721636    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4213395543a6ef339728f8523f33f15faf75bdee21587970d4f87f953cf526e"} err="failed to get container status \"a4213395543a6ef339728f8523f33f15faf75bdee21587970d4f87f953cf526e\": rpc error: code = NotFound desc = could not find container \"a4213395543a6ef339728f8523f33f15faf75bdee21587970d4f87f953cf526e\": container with ID starting with a4213395543a6ef339728f8523f33f15faf75bdee21587970d4f87f953cf526e not found: ID does not exist"
Oct 30 21:50:42.721691 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.721653    5678 scope.go:122] "RemoveContainer" containerID="ebbf6d346fb1ef77b512d07e571efc8794da78e6d86fc712a611858d3530ec5d"
Oct 30 21:50:42.721823 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.721808    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebbf6d346fb1ef77b512d07e571efc8794da78e6d86fc712a611858d3530ec5d"} err="failed to get container status \"ebbf6d346fb1ef77b512d07e571efc8794da78e6d86fc712a611858d3530ec5d\": rpc error: code = NotFound desc = could not find container \"ebbf6d346fb1ef77b512d07e571efc8794da78e6d86fc712a611858d3530ec5d\": container with ID starting with ebbf6d346fb1ef77b512d07e571efc8794da78e6d86fc712a611858d3530ec5d not found: ID does not exist"
Oct 30 21:50:42.721860 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.721823    5678 scope.go:122] "RemoveContainer" containerID="f7f271fecf3370b4e1a7117367120bc8e1a3ba6129d0a2171e8c002bee61190a"
Oct 30 21:50:42.721986 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.721972    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7f271fecf3370b4e1a7117367120bc8e1a3ba6129d0a2171e8c002bee61190a"} err="failed to get container status \"f7f271fecf3370b4e1a7117367120bc8e1a3ba6129d0a2171e8c002bee61190a\": rpc error: code = NotFound desc = could not find container \"f7f271fecf3370b4e1a7117367120bc8e1a3ba6129d0a2171e8c002bee61190a\": container with ID starting with f7f271fecf3370b4e1a7117367120bc8e1a3ba6129d0a2171e8c002bee61190a not found: ID does not exist"
Oct 30 21:50:42.722028 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.721986    5678 scope.go:122] "RemoveContainer" containerID="0026372bec6ebf4cc2f80b456ae15eafc47d5675ef85d041318db9169cdec013"
Oct 30 21:50:42.722149 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.722134    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0026372bec6ebf4cc2f80b456ae15eafc47d5675ef85d041318db9169cdec013"} err="failed to get container status \"0026372bec6ebf4cc2f80b456ae15eafc47d5675ef85d041318db9169cdec013\": rpc error: code = NotFound desc = could not find container \"0026372bec6ebf4cc2f80b456ae15eafc47d5675ef85d041318db9169cdec013\": container with ID starting with 0026372bec6ebf4cc2f80b456ae15eafc47d5675ef85d041318db9169cdec013 not found: ID does not exist"
Oct 30 21:50:42.722189 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.722148    5678 scope.go:122] "RemoveContainer" containerID="d2766685272356bcd2c86bf5bf3b1cce9d467808eba84132b88fbbe31d1643ef"
Oct 30 21:50:42.722310 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.722294    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2766685272356bcd2c86bf5bf3b1cce9d467808eba84132b88fbbe31d1643ef"} err="failed to get container status \"d2766685272356bcd2c86bf5bf3b1cce9d467808eba84132b88fbbe31d1643ef\": rpc error: code = NotFound desc = could not find container \"d2766685272356bcd2c86bf5bf3b1cce9d467808eba84132b88fbbe31d1643ef\": container with ID starting with d2766685272356bcd2c86bf5bf3b1cce9d467808eba84132b88fbbe31d1643ef not found: ID does not exist"
Oct 30 21:50:42.722347 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.722310    5678 scope.go:122] "RemoveContainer" containerID="4226791aeb98d543049bf57efc9d5e8b19e4cf53528ff2a742fda907deb4c396"
Oct 30 21:50:42.722542 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.722526    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4226791aeb98d543049bf57efc9d5e8b19e4cf53528ff2a742fda907deb4c396"} err="failed to get container status \"4226791aeb98d543049bf57efc9d5e8b19e4cf53528ff2a742fda907deb4c396\": rpc error: code = NotFound desc = could not find container \"4226791aeb98d543049bf57efc9d5e8b19e4cf53528ff2a742fda907deb4c396\": container with ID starting with 4226791aeb98d543049bf57efc9d5e8b19e4cf53528ff2a742fda907deb4c396 not found: ID does not exist"
Oct 30 21:50:42.722588 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.722541    5678 scope.go:122] "RemoveContainer" containerID="a93d15733e8cc09f6a51bfb22088505a1330edba1d3d0d6106aea54679c62b22"
Oct 30 21:50:42.722771 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.722751    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a93d15733e8cc09f6a51bfb22088505a1330edba1d3d0d6106aea54679c62b22"} err="failed to get container status \"a93d15733e8cc09f6a51bfb22088505a1330edba1d3d0d6106aea54679c62b22\": rpc error: code = NotFound desc = could not find container \"a93d15733e8cc09f6a51bfb22088505a1330edba1d3d0d6106aea54679c62b22\": container with ID starting with a93d15733e8cc09f6a51bfb22088505a1330edba1d3d0d6106aea54679c62b22 not found: ID does not exist"
Oct 30 21:50:42.722829 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.722771    5678 scope.go:122] "RemoveContainer" containerID="a4213395543a6ef339728f8523f33f15faf75bdee21587970d4f87f953cf526e"
Oct 30 21:50:42.722974 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.722959    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4213395543a6ef339728f8523f33f15faf75bdee21587970d4f87f953cf526e"} err="failed to get container status \"a4213395543a6ef339728f8523f33f15faf75bdee21587970d4f87f953cf526e\": rpc error: code = NotFound desc = could not find container \"a4213395543a6ef339728f8523f33f15faf75bdee21587970d4f87f953cf526e\": container with ID starting with a4213395543a6ef339728f8523f33f15faf75bdee21587970d4f87f953cf526e not found: ID does not exist"
Oct 30 21:50:42.723011 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.722974    5678 scope.go:122] "RemoveContainer" containerID="ebbf6d346fb1ef77b512d07e571efc8794da78e6d86fc712a611858d3530ec5d"
Oct 30 21:50:42.723180 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.723162    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebbf6d346fb1ef77b512d07e571efc8794da78e6d86fc712a611858d3530ec5d"} err="failed to get container status \"ebbf6d346fb1ef77b512d07e571efc8794da78e6d86fc712a611858d3530ec5d\": rpc error: code = NotFound desc = could not find container \"ebbf6d346fb1ef77b512d07e571efc8794da78e6d86fc712a611858d3530ec5d\": container with ID starting with ebbf6d346fb1ef77b512d07e571efc8794da78e6d86fc712a611858d3530ec5d not found: ID does not exist"
Oct 30 21:50:42.723216 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.723185    5678 scope.go:122] "RemoveContainer" containerID="f7f271fecf3370b4e1a7117367120bc8e1a3ba6129d0a2171e8c002bee61190a"
Oct 30 21:50:42.723374 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.723360    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7f271fecf3370b4e1a7117367120bc8e1a3ba6129d0a2171e8c002bee61190a"} err="failed to get container status \"f7f271fecf3370b4e1a7117367120bc8e1a3ba6129d0a2171e8c002bee61190a\": rpc error: code = NotFound desc = could not find container \"f7f271fecf3370b4e1a7117367120bc8e1a3ba6129d0a2171e8c002bee61190a\": container with ID starting with f7f271fecf3370b4e1a7117367120bc8e1a3ba6129d0a2171e8c002bee61190a not found: ID does not exist"
Oct 30 21:50:42.826817 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.826760    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 21:50:42.873988 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:42.873949    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 21:50:44.520349 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:44.520315    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5138fe8c-9927-4d05-8fe5-5556072ed595" path="/var/lib/kubelet/pods/5138fe8c-9927-4d05-8fe5-5556072ed595/volumes"
Oct 30 21:50:47.839615 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:47.839540    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-dc7c7d47d-ghdn7"
Oct 30 21:50:47.846699 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:47.846670    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-dc7c7d47d-ghdn7"
Oct 30 21:50:47.911475 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:47.911443    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 21:50:47.911852 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:47.911799    5678 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ef7ae922-6da7-4817-ba91-35e67b6ac8c5" containerName="prometheus" containerID="cri-o://29ceada24258c6d887579990630dc0a842a2ba2a018b98512da626fb7f4b54d9" gracePeriod=600
Oct 30 21:50:47.911933 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:47.911850    5678 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ef7ae922-6da7-4817-ba91-35e67b6ac8c5" containerName="kube-rbac-proxy" containerID="cri-o://1ff12d7276ddc77131dc926b364267852a917f820b7e54191df7b71e90f45ca1" gracePeriod=600
Oct 30 21:50:47.911997 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:47.911902    5678 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ef7ae922-6da7-4817-ba91-35e67b6ac8c5" containerName="kube-rbac-proxy-web" containerID="cri-o://b6e73fa7c904d0a775556c74faaf3620b284efffebf5489198770681155a28d0" gracePeriod=600
Oct 30 21:50:47.911997 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:47.911879    5678 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ef7ae922-6da7-4817-ba91-35e67b6ac8c5" containerName="config-reloader" containerID="cri-o://80f487ffb5dcf6fc6d14099c357fbc7e27472b4a78950699c63c53d90770256a" gracePeriod=600
Oct 30 21:50:47.912093 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:47.911991    5678 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ef7ae922-6da7-4817-ba91-35e67b6ac8c5" containerName="kube-rbac-proxy-thanos" containerID="cri-o://9c8be5ed7b18aa2995dfcea5a35d31fa9f057c30716bfa5b89553003d7cc68d2" gracePeriod=600
Oct 30 21:50:47.912186 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:47.912162    5678 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ef7ae922-6da7-4817-ba91-35e67b6ac8c5" containerName="thanos-sidecar" containerID="cri-o://7f01cffd7d46651522100fea81bb705ec595c21b7479821b5b352982cd7d4988" gracePeriod=600
Oct 30 21:50:48.645161 ip-10-0-128-245 kubenswrapper[5678]: E1030 21:50:48.645116    5678 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Canceled desc = context canceled" containerID="29ceada24258c6d887579990630dc0a842a2ba2a018b98512da626fb7f4b54d9" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"]
Oct 30 21:50:48.645337 ip-10-0-128-245 kubenswrapper[5678]: E1030 21:50:48.645192    5678 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Canceled desc = context canceled" containerID="29ceada24258c6d887579990630dc0a842a2ba2a018b98512da626fb7f4b54d9" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"]
Oct 30 21:50:48.645337 ip-10-0-128-245 kubenswrapper[5678]: E1030 21:50:48.645216    5678 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Canceled desc = context canceled" containerID="29ceada24258c6d887579990630dc0a842a2ba2a018b98512da626fb7f4b54d9" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"]
Oct 30 21:50:48.645337 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.645235    5678 prober.go:106] "Probe errored" probeType="Readiness" pod="openshift-monitoring/prometheus-k8s-0" podUID="ef7ae922-6da7-4817-ba91-35e67b6ac8c5" containerName="prometheus" probeResult="unknown" err="rpc error: code = Canceled desc = context canceled"
Oct 30 21:50:48.660311 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.660282    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:48.691116 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.691089    5678 generic.go:356] "Generic (PLEG): container finished" podID="ef7ae922-6da7-4817-ba91-35e67b6ac8c5" containerID="9c8be5ed7b18aa2995dfcea5a35d31fa9f057c30716bfa5b89553003d7cc68d2" exitCode=0
Oct 30 21:50:48.691116 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.691115    5678 generic.go:356] "Generic (PLEG): container finished" podID="ef7ae922-6da7-4817-ba91-35e67b6ac8c5" containerID="1ff12d7276ddc77131dc926b364267852a917f820b7e54191df7b71e90f45ca1" exitCode=0
Oct 30 21:50:48.691284 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.691127    5678 generic.go:356] "Generic (PLEG): container finished" podID="ef7ae922-6da7-4817-ba91-35e67b6ac8c5" containerID="b6e73fa7c904d0a775556c74faaf3620b284efffebf5489198770681155a28d0" exitCode=0
Oct 30 21:50:48.691284 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.691135    5678 generic.go:356] "Generic (PLEG): container finished" podID="ef7ae922-6da7-4817-ba91-35e67b6ac8c5" containerID="7f01cffd7d46651522100fea81bb705ec595c21b7479821b5b352982cd7d4988" exitCode=0
Oct 30 21:50:48.691284 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.691145    5678 generic.go:356] "Generic (PLEG): container finished" podID="ef7ae922-6da7-4817-ba91-35e67b6ac8c5" containerID="80f487ffb5dcf6fc6d14099c357fbc7e27472b4a78950699c63c53d90770256a" exitCode=0
Oct 30 21:50:48.691284 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.691152    5678 generic.go:356] "Generic (PLEG): container finished" podID="ef7ae922-6da7-4817-ba91-35e67b6ac8c5" containerID="29ceada24258c6d887579990630dc0a842a2ba2a018b98512da626fb7f4b54d9" exitCode=0
Oct 30 21:50:48.691284 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.691160    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:48.691284 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.691162    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ef7ae922-6da7-4817-ba91-35e67b6ac8c5","Type":"ContainerDied","Data":"9c8be5ed7b18aa2995dfcea5a35d31fa9f057c30716bfa5b89553003d7cc68d2"}
Oct 30 21:50:48.691284 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.691251    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ef7ae922-6da7-4817-ba91-35e67b6ac8c5","Type":"ContainerDied","Data":"1ff12d7276ddc77131dc926b364267852a917f820b7e54191df7b71e90f45ca1"}
Oct 30 21:50:48.691284 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.691278    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ef7ae922-6da7-4817-ba91-35e67b6ac8c5","Type":"ContainerDied","Data":"b6e73fa7c904d0a775556c74faaf3620b284efffebf5489198770681155a28d0"}
Oct 30 21:50:48.691571 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.691302    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ef7ae922-6da7-4817-ba91-35e67b6ac8c5","Type":"ContainerDied","Data":"7f01cffd7d46651522100fea81bb705ec595c21b7479821b5b352982cd7d4988"}
Oct 30 21:50:48.691571 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.691326    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ef7ae922-6da7-4817-ba91-35e67b6ac8c5","Type":"ContainerDied","Data":"80f487ffb5dcf6fc6d14099c357fbc7e27472b4a78950699c63c53d90770256a"}
Oct 30 21:50:48.691571 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.691354    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ef7ae922-6da7-4817-ba91-35e67b6ac8c5","Type":"ContainerDied","Data":"29ceada24258c6d887579990630dc0a842a2ba2a018b98512da626fb7f4b54d9"}
Oct 30 21:50:48.691571 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.691376    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ef7ae922-6da7-4817-ba91-35e67b6ac8c5","Type":"ContainerDied","Data":"266940a224ea0a54c1a8a2d9b4e5b33f7b149ab8f92cc7b373d4259ede742beb"}
Oct 30 21:50:48.691571 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.691325    5678 scope.go:122] "RemoveContainer" containerID="9c8be5ed7b18aa2995dfcea5a35d31fa9f057c30716bfa5b89553003d7cc68d2"
Oct 30 21:50:48.698348 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.698331    5678 scope.go:122] "RemoveContainer" containerID="1ff12d7276ddc77131dc926b364267852a917f820b7e54191df7b71e90f45ca1"
Oct 30 21:50:48.704743 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.704726    5678 scope.go:122] "RemoveContainer" containerID="b6e73fa7c904d0a775556c74faaf3620b284efffebf5489198770681155a28d0"
Oct 30 21:50:48.711325 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.711302    5678 scope.go:122] "RemoveContainer" containerID="7f01cffd7d46651522100fea81bb705ec595c21b7479821b5b352982cd7d4988"
Oct 30 21:50:48.715360 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.715341    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-web-config\" (UniqueName: \"kubernetes.io/secret/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-web-config\") pod \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\" (UID: \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\") "
Oct 30 21:50:48.715452 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.715370    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-prometheus-k8s-db\") pod \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\" (UID: \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\") "
Oct 30 21:50:48.715452 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.715391    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-configmap-kubelet-serving-ca-bundle\") pod \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\" (UID: \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\") "
Oct 30 21:50:48.715452 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.715432    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-thanos-prometheus-http-client-file\") pod \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\" (UID: \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\") "
Oct 30 21:50:48.715622 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.715456    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\" (UID: \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\") "
Oct 30 21:50:48.715622 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.715559    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-tls-assets\" (UniqueName: \"kubernetes.io/projected/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-tls-assets\") pod \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\" (UID: \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\") "
Oct 30 21:50:48.715622 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.715605    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-config-out\" (UniqueName: \"kubernetes.io/empty-dir/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-config-out\") pod \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\" (UID: \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\") "
Oct 30 21:50:48.715768 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.715627    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-config\" (UniqueName: \"kubernetes.io/secret/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-config\") pod \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\" (UID: \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\") "
Oct 30 21:50:48.715768 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.715656    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-prometheus-k8s-rulefiles-2\") pod \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\" (UID: \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\") "
Oct 30 21:50:48.715768 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.715686    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-prometheus-k8s-rulefiles-0\") pod \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\" (UID: \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\") "
Oct 30 21:50:48.715768 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.715728    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-prometheus-k8s-rulefiles-1\") pod \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\" (UID: \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\") "
Oct 30 21:50:48.715768 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.715762    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\" (UID: \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\") "
Oct 30 21:50:48.716143 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.715805    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-secret-prometheus-k8s-tls\") pod \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\" (UID: \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\") "
Oct 30 21:50:48.716143 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.715836    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-kube-api-access-cr8x2\" (UniqueName: \"kubernetes.io/projected/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-kube-api-access-cr8x2\") pod \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\" (UID: \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\") "
Oct 30 21:50:48.716143 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.715830    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-configmap-kubelet-serving-ca-bundle" pod "ef7ae922-6da7-4817-ba91-35e67b6ac8c5" (UID: "ef7ae922-6da7-4817-ba91-35e67b6ac8c5"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:50:48.716143 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.715867    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-secret-metrics-client-certs\") pod \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\" (UID: \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\") "
Oct 30 21:50:48.716143 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.715908    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-prometheus-trusted-ca-bundle\") pod \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\" (UID: \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\") "
Oct 30 21:50:48.716143 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.715937    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-configmap-metrics-client-ca\") pod \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\" (UID: \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\") "
Oct 30 21:50:48.716143 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.715972    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-secret-grpc-tls\") pod \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\" (UID: \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\") "
Oct 30 21:50:48.716143 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.716009    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-secret-kube-rbac-proxy\") pod \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\" (UID: \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\") "
Oct 30 21:50:48.716143 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.716045    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-configmap-serving-certs-ca-bundle\") pod \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\" (UID: \"ef7ae922-6da7-4817-ba91-35e67b6ac8c5\") "
Oct 30 21:50:48.716143 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.716129    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-prometheus-k8s-db" pod "ef7ae922-6da7-4817-ba91-35e67b6ac8c5" (UID: "ef7ae922-6da7-4817-ba91-35e67b6ac8c5"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:50:48.716825 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.716361    5678 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-prometheus-k8s-db\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:50:48.716825 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.716386    5678 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:50:48.716825 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.716533    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-configmap-serving-certs-ca-bundle" pod "ef7ae922-6da7-4817-ba91-35e67b6ac8c5" (UID: "ef7ae922-6da7-4817-ba91-35e67b6ac8c5"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:50:48.716825 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.716770    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-prometheus-k8s-rulefiles-2" pod "ef7ae922-6da7-4817-ba91-35e67b6ac8c5" (UID: "ef7ae922-6da7-4817-ba91-35e67b6ac8c5"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:50:48.717033 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.717008    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-prometheus-trusted-ca-bundle" pod "ef7ae922-6da7-4817-ba91-35e67b6ac8c5" (UID: "ef7ae922-6da7-4817-ba91-35e67b6ac8c5"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:50:48.717536 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.717510    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-prometheus-k8s-rulefiles-1" pod "ef7ae922-6da7-4817-ba91-35e67b6ac8c5" (UID: "ef7ae922-6da7-4817-ba91-35e67b6ac8c5"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:50:48.717921 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.717896    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-configmap-metrics-client-ca" pod "ef7ae922-6da7-4817-ba91-35e67b6ac8c5" (UID: "ef7ae922-6da7-4817-ba91-35e67b6ac8c5"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:50:48.718599 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.718574    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-prometheus-k8s-rulefiles-0" pod "ef7ae922-6da7-4817-ba91-35e67b6ac8c5" (UID: "ef7ae922-6da7-4817-ba91-35e67b6ac8c5"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:50:48.720051 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.720033    5678 scope.go:122] "RemoveContainer" containerID="80f487ffb5dcf6fc6d14099c357fbc7e27472b4a78950699c63c53d90770256a"
Oct 30 21:50:48.720159 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.720136    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-config" pod "ef7ae922-6da7-4817-ba91-35e67b6ac8c5" (UID: "ef7ae922-6da7-4817-ba91-35e67b6ac8c5"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:50:48.720268 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.720227    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-secret-prometheus-k8s-tls" pod "ef7ae922-6da7-4817-ba91-35e67b6ac8c5" (UID: "ef7ae922-6da7-4817-ba91-35e67b6ac8c5"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:50:48.720731 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.720528    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-secret-prometheus-k8s-thanos-sidecar-tls" pod "ef7ae922-6da7-4817-ba91-35e67b6ac8c5" (UID: "ef7ae922-6da7-4817-ba91-35e67b6ac8c5"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:50:48.720731 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.720585    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-kube-api-access-cr8x2" pod "ef7ae922-6da7-4817-ba91-35e67b6ac8c5" (UID: "ef7ae922-6da7-4817-ba91-35e67b6ac8c5"). InnerVolumeSpecName "kube-api-access-cr8x2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:50:48.721534 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.721508    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-config-out" pod "ef7ae922-6da7-4817-ba91-35e67b6ac8c5" (UID: "ef7ae922-6da7-4817-ba91-35e67b6ac8c5"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:50:48.721627 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.721531    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-thanos-prometheus-http-client-file" pod "ef7ae922-6da7-4817-ba91-35e67b6ac8c5" (UID: "ef7ae922-6da7-4817-ba91-35e67b6ac8c5"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:50:48.721627 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.721580    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-tls-assets" pod "ef7ae922-6da7-4817-ba91-35e67b6ac8c5" (UID: "ef7ae922-6da7-4817-ba91-35e67b6ac8c5"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:50:48.721627 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.721604    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-secret-prometheus-k8s-kube-rbac-proxy-web" pod "ef7ae922-6da7-4817-ba91-35e67b6ac8c5" (UID: "ef7ae922-6da7-4817-ba91-35e67b6ac8c5"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:50:48.721726 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.721674    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-secret-grpc-tls" pod "ef7ae922-6da7-4817-ba91-35e67b6ac8c5" (UID: "ef7ae922-6da7-4817-ba91-35e67b6ac8c5"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:50:48.722264 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.722237    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-secret-kube-rbac-proxy" pod "ef7ae922-6da7-4817-ba91-35e67b6ac8c5" (UID: "ef7ae922-6da7-4817-ba91-35e67b6ac8c5"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:50:48.722707 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.722690    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-secret-metrics-client-certs" pod "ef7ae922-6da7-4817-ba91-35e67b6ac8c5" (UID: "ef7ae922-6da7-4817-ba91-35e67b6ac8c5"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:50:48.727236 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.727220    5678 scope.go:122] "RemoveContainer" containerID="29ceada24258c6d887579990630dc0a842a2ba2a018b98512da626fb7f4b54d9"
Oct 30 21:50:48.730080 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.730058    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-web-config" pod "ef7ae922-6da7-4817-ba91-35e67b6ac8c5" (UID: "ef7ae922-6da7-4817-ba91-35e67b6ac8c5"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:50:48.734135 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.734118    5678 scope.go:122] "RemoveContainer" containerID="258fd66deee3797d90e92c40203fe9ebf6228e43b399cf1d0332b8034deed533"
Oct 30 21:50:48.740258 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.740242    5678 scope.go:122] "RemoveContainer" containerID="9c8be5ed7b18aa2995dfcea5a35d31fa9f057c30716bfa5b89553003d7cc68d2"
Oct 30 21:50:48.740541 ip-10-0-128-245 kubenswrapper[5678]: E1030 21:50:48.740521    5678 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c8be5ed7b18aa2995dfcea5a35d31fa9f057c30716bfa5b89553003d7cc68d2\": container with ID starting with 9c8be5ed7b18aa2995dfcea5a35d31fa9f057c30716bfa5b89553003d7cc68d2 not found: ID does not exist" containerID="9c8be5ed7b18aa2995dfcea5a35d31fa9f057c30716bfa5b89553003d7cc68d2"
Oct 30 21:50:48.740607 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.740549    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c8be5ed7b18aa2995dfcea5a35d31fa9f057c30716bfa5b89553003d7cc68d2"} err="failed to get container status \"9c8be5ed7b18aa2995dfcea5a35d31fa9f057c30716bfa5b89553003d7cc68d2\": rpc error: code = NotFound desc = could not find container \"9c8be5ed7b18aa2995dfcea5a35d31fa9f057c30716bfa5b89553003d7cc68d2\": container with ID starting with 9c8be5ed7b18aa2995dfcea5a35d31fa9f057c30716bfa5b89553003d7cc68d2 not found: ID does not exist"
Oct 30 21:50:48.740607 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.740565    5678 scope.go:122] "RemoveContainer" containerID="1ff12d7276ddc77131dc926b364267852a917f820b7e54191df7b71e90f45ca1"
Oct 30 21:50:48.740795 ip-10-0-128-245 kubenswrapper[5678]: E1030 21:50:48.740779    5678 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ff12d7276ddc77131dc926b364267852a917f820b7e54191df7b71e90f45ca1\": container with ID starting with 1ff12d7276ddc77131dc926b364267852a917f820b7e54191df7b71e90f45ca1 not found: ID does not exist" containerID="1ff12d7276ddc77131dc926b364267852a917f820b7e54191df7b71e90f45ca1"
Oct 30 21:50:48.740841 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.740799    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ff12d7276ddc77131dc926b364267852a917f820b7e54191df7b71e90f45ca1"} err="failed to get container status \"1ff12d7276ddc77131dc926b364267852a917f820b7e54191df7b71e90f45ca1\": rpc error: code = NotFound desc = could not find container \"1ff12d7276ddc77131dc926b364267852a917f820b7e54191df7b71e90f45ca1\": container with ID starting with 1ff12d7276ddc77131dc926b364267852a917f820b7e54191df7b71e90f45ca1 not found: ID does not exist"
Oct 30 21:50:48.740841 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.740812    5678 scope.go:122] "RemoveContainer" containerID="b6e73fa7c904d0a775556c74faaf3620b284efffebf5489198770681155a28d0"
Oct 30 21:50:48.741037 ip-10-0-128-245 kubenswrapper[5678]: E1030 21:50:48.741015    5678 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6e73fa7c904d0a775556c74faaf3620b284efffebf5489198770681155a28d0\": container with ID starting with b6e73fa7c904d0a775556c74faaf3620b284efffebf5489198770681155a28d0 not found: ID does not exist" containerID="b6e73fa7c904d0a775556c74faaf3620b284efffebf5489198770681155a28d0"
Oct 30 21:50:48.741103 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.741044    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6e73fa7c904d0a775556c74faaf3620b284efffebf5489198770681155a28d0"} err="failed to get container status \"b6e73fa7c904d0a775556c74faaf3620b284efffebf5489198770681155a28d0\": rpc error: code = NotFound desc = could not find container \"b6e73fa7c904d0a775556c74faaf3620b284efffebf5489198770681155a28d0\": container with ID starting with b6e73fa7c904d0a775556c74faaf3620b284efffebf5489198770681155a28d0 not found: ID does not exist"
Oct 30 21:50:48.741103 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.741059    5678 scope.go:122] "RemoveContainer" containerID="7f01cffd7d46651522100fea81bb705ec595c21b7479821b5b352982cd7d4988"
Oct 30 21:50:48.741285 ip-10-0-128-245 kubenswrapper[5678]: E1030 21:50:48.741270    5678 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f01cffd7d46651522100fea81bb705ec595c21b7479821b5b352982cd7d4988\": container with ID starting with 7f01cffd7d46651522100fea81bb705ec595c21b7479821b5b352982cd7d4988 not found: ID does not exist" containerID="7f01cffd7d46651522100fea81bb705ec595c21b7479821b5b352982cd7d4988"
Oct 30 21:50:48.741331 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.741288    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f01cffd7d46651522100fea81bb705ec595c21b7479821b5b352982cd7d4988"} err="failed to get container status \"7f01cffd7d46651522100fea81bb705ec595c21b7479821b5b352982cd7d4988\": rpc error: code = NotFound desc = could not find container \"7f01cffd7d46651522100fea81bb705ec595c21b7479821b5b352982cd7d4988\": container with ID starting with 7f01cffd7d46651522100fea81bb705ec595c21b7479821b5b352982cd7d4988 not found: ID does not exist"
Oct 30 21:50:48.741331 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.741299    5678 scope.go:122] "RemoveContainer" containerID="80f487ffb5dcf6fc6d14099c357fbc7e27472b4a78950699c63c53d90770256a"
Oct 30 21:50:48.741534 ip-10-0-128-245 kubenswrapper[5678]: E1030 21:50:48.741519    5678 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80f487ffb5dcf6fc6d14099c357fbc7e27472b4a78950699c63c53d90770256a\": container with ID starting with 80f487ffb5dcf6fc6d14099c357fbc7e27472b4a78950699c63c53d90770256a not found: ID does not exist" containerID="80f487ffb5dcf6fc6d14099c357fbc7e27472b4a78950699c63c53d90770256a"
Oct 30 21:50:48.741582 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.741537    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80f487ffb5dcf6fc6d14099c357fbc7e27472b4a78950699c63c53d90770256a"} err="failed to get container status \"80f487ffb5dcf6fc6d14099c357fbc7e27472b4a78950699c63c53d90770256a\": rpc error: code = NotFound desc = could not find container \"80f487ffb5dcf6fc6d14099c357fbc7e27472b4a78950699c63c53d90770256a\": container with ID starting with 80f487ffb5dcf6fc6d14099c357fbc7e27472b4a78950699c63c53d90770256a not found: ID does not exist"
Oct 30 21:50:48.741582 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.741551    5678 scope.go:122] "RemoveContainer" containerID="29ceada24258c6d887579990630dc0a842a2ba2a018b98512da626fb7f4b54d9"
Oct 30 21:50:48.741768 ip-10-0-128-245 kubenswrapper[5678]: E1030 21:50:48.741754    5678 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29ceada24258c6d887579990630dc0a842a2ba2a018b98512da626fb7f4b54d9\": container with ID starting with 29ceada24258c6d887579990630dc0a842a2ba2a018b98512da626fb7f4b54d9 not found: ID does not exist" containerID="29ceada24258c6d887579990630dc0a842a2ba2a018b98512da626fb7f4b54d9"
Oct 30 21:50:48.741816 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.741771    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29ceada24258c6d887579990630dc0a842a2ba2a018b98512da626fb7f4b54d9"} err="failed to get container status \"29ceada24258c6d887579990630dc0a842a2ba2a018b98512da626fb7f4b54d9\": rpc error: code = NotFound desc = could not find container \"29ceada24258c6d887579990630dc0a842a2ba2a018b98512da626fb7f4b54d9\": container with ID starting with 29ceada24258c6d887579990630dc0a842a2ba2a018b98512da626fb7f4b54d9 not found: ID does not exist"
Oct 30 21:50:48.741816 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.741784    5678 scope.go:122] "RemoveContainer" containerID="258fd66deee3797d90e92c40203fe9ebf6228e43b399cf1d0332b8034deed533"
Oct 30 21:50:48.742008 ip-10-0-128-245 kubenswrapper[5678]: E1030 21:50:48.741994    5678 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"258fd66deee3797d90e92c40203fe9ebf6228e43b399cf1d0332b8034deed533\": container with ID starting with 258fd66deee3797d90e92c40203fe9ebf6228e43b399cf1d0332b8034deed533 not found: ID does not exist" containerID="258fd66deee3797d90e92c40203fe9ebf6228e43b399cf1d0332b8034deed533"
Oct 30 21:50:48.742054 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.742010    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"258fd66deee3797d90e92c40203fe9ebf6228e43b399cf1d0332b8034deed533"} err="failed to get container status \"258fd66deee3797d90e92c40203fe9ebf6228e43b399cf1d0332b8034deed533\": rpc error: code = NotFound desc = could not find container \"258fd66deee3797d90e92c40203fe9ebf6228e43b399cf1d0332b8034deed533\": container with ID starting with 258fd66deee3797d90e92c40203fe9ebf6228e43b399cf1d0332b8034deed533 not found: ID does not exist"
Oct 30 21:50:48.742054 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.742021    5678 scope.go:122] "RemoveContainer" containerID="9c8be5ed7b18aa2995dfcea5a35d31fa9f057c30716bfa5b89553003d7cc68d2"
Oct 30 21:50:48.742248 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.742232    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c8be5ed7b18aa2995dfcea5a35d31fa9f057c30716bfa5b89553003d7cc68d2"} err="failed to get container status \"9c8be5ed7b18aa2995dfcea5a35d31fa9f057c30716bfa5b89553003d7cc68d2\": rpc error: code = NotFound desc = could not find container \"9c8be5ed7b18aa2995dfcea5a35d31fa9f057c30716bfa5b89553003d7cc68d2\": container with ID starting with 9c8be5ed7b18aa2995dfcea5a35d31fa9f057c30716bfa5b89553003d7cc68d2 not found: ID does not exist"
Oct 30 21:50:48.742248 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.742247    5678 scope.go:122] "RemoveContainer" containerID="1ff12d7276ddc77131dc926b364267852a917f820b7e54191df7b71e90f45ca1"
Oct 30 21:50:48.742482 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.742466    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ff12d7276ddc77131dc926b364267852a917f820b7e54191df7b71e90f45ca1"} err="failed to get container status \"1ff12d7276ddc77131dc926b364267852a917f820b7e54191df7b71e90f45ca1\": rpc error: code = NotFound desc = could not find container \"1ff12d7276ddc77131dc926b364267852a917f820b7e54191df7b71e90f45ca1\": container with ID starting with 1ff12d7276ddc77131dc926b364267852a917f820b7e54191df7b71e90f45ca1 not found: ID does not exist"
Oct 30 21:50:48.742482 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.742481    5678 scope.go:122] "RemoveContainer" containerID="b6e73fa7c904d0a775556c74faaf3620b284efffebf5489198770681155a28d0"
Oct 30 21:50:48.742689 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.742672    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6e73fa7c904d0a775556c74faaf3620b284efffebf5489198770681155a28d0"} err="failed to get container status \"b6e73fa7c904d0a775556c74faaf3620b284efffebf5489198770681155a28d0\": rpc error: code = NotFound desc = could not find container \"b6e73fa7c904d0a775556c74faaf3620b284efffebf5489198770681155a28d0\": container with ID starting with b6e73fa7c904d0a775556c74faaf3620b284efffebf5489198770681155a28d0 not found: ID does not exist"
Oct 30 21:50:48.742689 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.742687    5678 scope.go:122] "RemoveContainer" containerID="7f01cffd7d46651522100fea81bb705ec595c21b7479821b5b352982cd7d4988"
Oct 30 21:50:48.742901 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.742884    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f01cffd7d46651522100fea81bb705ec595c21b7479821b5b352982cd7d4988"} err="failed to get container status \"7f01cffd7d46651522100fea81bb705ec595c21b7479821b5b352982cd7d4988\": rpc error: code = NotFound desc = could not find container \"7f01cffd7d46651522100fea81bb705ec595c21b7479821b5b352982cd7d4988\": container with ID starting with 7f01cffd7d46651522100fea81bb705ec595c21b7479821b5b352982cd7d4988 not found: ID does not exist"
Oct 30 21:50:48.742938 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.742900    5678 scope.go:122] "RemoveContainer" containerID="80f487ffb5dcf6fc6d14099c357fbc7e27472b4a78950699c63c53d90770256a"
Oct 30 21:50:48.743098 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.743080    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80f487ffb5dcf6fc6d14099c357fbc7e27472b4a78950699c63c53d90770256a"} err="failed to get container status \"80f487ffb5dcf6fc6d14099c357fbc7e27472b4a78950699c63c53d90770256a\": rpc error: code = NotFound desc = could not find container \"80f487ffb5dcf6fc6d14099c357fbc7e27472b4a78950699c63c53d90770256a\": container with ID starting with 80f487ffb5dcf6fc6d14099c357fbc7e27472b4a78950699c63c53d90770256a not found: ID does not exist"
Oct 30 21:50:48.743139 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.743099    5678 scope.go:122] "RemoveContainer" containerID="29ceada24258c6d887579990630dc0a842a2ba2a018b98512da626fb7f4b54d9"
Oct 30 21:50:48.743274 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.743260    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29ceada24258c6d887579990630dc0a842a2ba2a018b98512da626fb7f4b54d9"} err="failed to get container status \"29ceada24258c6d887579990630dc0a842a2ba2a018b98512da626fb7f4b54d9\": rpc error: code = NotFound desc = could not find container \"29ceada24258c6d887579990630dc0a842a2ba2a018b98512da626fb7f4b54d9\": container with ID starting with 29ceada24258c6d887579990630dc0a842a2ba2a018b98512da626fb7f4b54d9 not found: ID does not exist"
Oct 30 21:50:48.743324 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.743274    5678 scope.go:122] "RemoveContainer" containerID="258fd66deee3797d90e92c40203fe9ebf6228e43b399cf1d0332b8034deed533"
Oct 30 21:50:48.743496 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.743479    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"258fd66deee3797d90e92c40203fe9ebf6228e43b399cf1d0332b8034deed533"} err="failed to get container status \"258fd66deee3797d90e92c40203fe9ebf6228e43b399cf1d0332b8034deed533\": rpc error: code = NotFound desc = could not find container \"258fd66deee3797d90e92c40203fe9ebf6228e43b399cf1d0332b8034deed533\": container with ID starting with 258fd66deee3797d90e92c40203fe9ebf6228e43b399cf1d0332b8034deed533 not found: ID does not exist"
Oct 30 21:50:48.743496 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.743495    5678 scope.go:122] "RemoveContainer" containerID="9c8be5ed7b18aa2995dfcea5a35d31fa9f057c30716bfa5b89553003d7cc68d2"
Oct 30 21:50:48.743664 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.743649    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c8be5ed7b18aa2995dfcea5a35d31fa9f057c30716bfa5b89553003d7cc68d2"} err="failed to get container status \"9c8be5ed7b18aa2995dfcea5a35d31fa9f057c30716bfa5b89553003d7cc68d2\": rpc error: code = NotFound desc = could not find container \"9c8be5ed7b18aa2995dfcea5a35d31fa9f057c30716bfa5b89553003d7cc68d2\": container with ID starting with 9c8be5ed7b18aa2995dfcea5a35d31fa9f057c30716bfa5b89553003d7cc68d2 not found: ID does not exist"
Oct 30 21:50:48.743705 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.743664    5678 scope.go:122] "RemoveContainer" containerID="1ff12d7276ddc77131dc926b364267852a917f820b7e54191df7b71e90f45ca1"
Oct 30 21:50:48.743859 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.743843    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ff12d7276ddc77131dc926b364267852a917f820b7e54191df7b71e90f45ca1"} err="failed to get container status \"1ff12d7276ddc77131dc926b364267852a917f820b7e54191df7b71e90f45ca1\": rpc error: code = NotFound desc = could not find container \"1ff12d7276ddc77131dc926b364267852a917f820b7e54191df7b71e90f45ca1\": container with ID starting with 1ff12d7276ddc77131dc926b364267852a917f820b7e54191df7b71e90f45ca1 not found: ID does not exist"
Oct 30 21:50:48.743859 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.743857    5678 scope.go:122] "RemoveContainer" containerID="b6e73fa7c904d0a775556c74faaf3620b284efffebf5489198770681155a28d0"
Oct 30 21:50:48.744063 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.744047    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6e73fa7c904d0a775556c74faaf3620b284efffebf5489198770681155a28d0"} err="failed to get container status \"b6e73fa7c904d0a775556c74faaf3620b284efffebf5489198770681155a28d0\": rpc error: code = NotFound desc = could not find container \"b6e73fa7c904d0a775556c74faaf3620b284efffebf5489198770681155a28d0\": container with ID starting with b6e73fa7c904d0a775556c74faaf3620b284efffebf5489198770681155a28d0 not found: ID does not exist"
Oct 30 21:50:48.744118 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.744063    5678 scope.go:122] "RemoveContainer" containerID="7f01cffd7d46651522100fea81bb705ec595c21b7479821b5b352982cd7d4988"
Oct 30 21:50:48.744300 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.744284    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f01cffd7d46651522100fea81bb705ec595c21b7479821b5b352982cd7d4988"} err="failed to get container status \"7f01cffd7d46651522100fea81bb705ec595c21b7479821b5b352982cd7d4988\": rpc error: code = NotFound desc = could not find container \"7f01cffd7d46651522100fea81bb705ec595c21b7479821b5b352982cd7d4988\": container with ID starting with 7f01cffd7d46651522100fea81bb705ec595c21b7479821b5b352982cd7d4988 not found: ID does not exist"
Oct 30 21:50:48.744345 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.744300    5678 scope.go:122] "RemoveContainer" containerID="80f487ffb5dcf6fc6d14099c357fbc7e27472b4a78950699c63c53d90770256a"
Oct 30 21:50:48.744545 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.744528    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80f487ffb5dcf6fc6d14099c357fbc7e27472b4a78950699c63c53d90770256a"} err="failed to get container status \"80f487ffb5dcf6fc6d14099c357fbc7e27472b4a78950699c63c53d90770256a\": rpc error: code = NotFound desc = could not find container \"80f487ffb5dcf6fc6d14099c357fbc7e27472b4a78950699c63c53d90770256a\": container with ID starting with 80f487ffb5dcf6fc6d14099c357fbc7e27472b4a78950699c63c53d90770256a not found: ID does not exist"
Oct 30 21:50:48.744609 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.744545    5678 scope.go:122] "RemoveContainer" containerID="29ceada24258c6d887579990630dc0a842a2ba2a018b98512da626fb7f4b54d9"
Oct 30 21:50:48.744726 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.744710    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29ceada24258c6d887579990630dc0a842a2ba2a018b98512da626fb7f4b54d9"} err="failed to get container status \"29ceada24258c6d887579990630dc0a842a2ba2a018b98512da626fb7f4b54d9\": rpc error: code = NotFound desc = could not find container \"29ceada24258c6d887579990630dc0a842a2ba2a018b98512da626fb7f4b54d9\": container with ID starting with 29ceada24258c6d887579990630dc0a842a2ba2a018b98512da626fb7f4b54d9 not found: ID does not exist"
Oct 30 21:50:48.744763 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.744726    5678 scope.go:122] "RemoveContainer" containerID="258fd66deee3797d90e92c40203fe9ebf6228e43b399cf1d0332b8034deed533"
Oct 30 21:50:48.744905 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.744888    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"258fd66deee3797d90e92c40203fe9ebf6228e43b399cf1d0332b8034deed533"} err="failed to get container status \"258fd66deee3797d90e92c40203fe9ebf6228e43b399cf1d0332b8034deed533\": rpc error: code = NotFound desc = could not find container \"258fd66deee3797d90e92c40203fe9ebf6228e43b399cf1d0332b8034deed533\": container with ID starting with 258fd66deee3797d90e92c40203fe9ebf6228e43b399cf1d0332b8034deed533 not found: ID does not exist"
Oct 30 21:50:48.744942 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.744906    5678 scope.go:122] "RemoveContainer" containerID="9c8be5ed7b18aa2995dfcea5a35d31fa9f057c30716bfa5b89553003d7cc68d2"
Oct 30 21:50:48.745126 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.745107    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c8be5ed7b18aa2995dfcea5a35d31fa9f057c30716bfa5b89553003d7cc68d2"} err="failed to get container status \"9c8be5ed7b18aa2995dfcea5a35d31fa9f057c30716bfa5b89553003d7cc68d2\": rpc error: code = NotFound desc = could not find container \"9c8be5ed7b18aa2995dfcea5a35d31fa9f057c30716bfa5b89553003d7cc68d2\": container with ID starting with 9c8be5ed7b18aa2995dfcea5a35d31fa9f057c30716bfa5b89553003d7cc68d2 not found: ID does not exist"
Oct 30 21:50:48.745193 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.745128    5678 scope.go:122] "RemoveContainer" containerID="1ff12d7276ddc77131dc926b364267852a917f820b7e54191df7b71e90f45ca1"
Oct 30 21:50:48.745340 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.745323    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ff12d7276ddc77131dc926b364267852a917f820b7e54191df7b71e90f45ca1"} err="failed to get container status \"1ff12d7276ddc77131dc926b364267852a917f820b7e54191df7b71e90f45ca1\": rpc error: code = NotFound desc = could not find container \"1ff12d7276ddc77131dc926b364267852a917f820b7e54191df7b71e90f45ca1\": container with ID starting with 1ff12d7276ddc77131dc926b364267852a917f820b7e54191df7b71e90f45ca1 not found: ID does not exist"
Oct 30 21:50:48.745379 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.745341    5678 scope.go:122] "RemoveContainer" containerID="b6e73fa7c904d0a775556c74faaf3620b284efffebf5489198770681155a28d0"
Oct 30 21:50:48.745591 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.745574    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6e73fa7c904d0a775556c74faaf3620b284efffebf5489198770681155a28d0"} err="failed to get container status \"b6e73fa7c904d0a775556c74faaf3620b284efffebf5489198770681155a28d0\": rpc error: code = NotFound desc = could not find container \"b6e73fa7c904d0a775556c74faaf3620b284efffebf5489198770681155a28d0\": container with ID starting with b6e73fa7c904d0a775556c74faaf3620b284efffebf5489198770681155a28d0 not found: ID does not exist"
Oct 30 21:50:48.745663 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.745592    5678 scope.go:122] "RemoveContainer" containerID="7f01cffd7d46651522100fea81bb705ec595c21b7479821b5b352982cd7d4988"
Oct 30 21:50:48.745829 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.745812    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f01cffd7d46651522100fea81bb705ec595c21b7479821b5b352982cd7d4988"} err="failed to get container status \"7f01cffd7d46651522100fea81bb705ec595c21b7479821b5b352982cd7d4988\": rpc error: code = NotFound desc = could not find container \"7f01cffd7d46651522100fea81bb705ec595c21b7479821b5b352982cd7d4988\": container with ID starting with 7f01cffd7d46651522100fea81bb705ec595c21b7479821b5b352982cd7d4988 not found: ID does not exist"
Oct 30 21:50:48.745874 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.745830    5678 scope.go:122] "RemoveContainer" containerID="80f487ffb5dcf6fc6d14099c357fbc7e27472b4a78950699c63c53d90770256a"
Oct 30 21:50:48.746030 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.746013    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80f487ffb5dcf6fc6d14099c357fbc7e27472b4a78950699c63c53d90770256a"} err="failed to get container status \"80f487ffb5dcf6fc6d14099c357fbc7e27472b4a78950699c63c53d90770256a\": rpc error: code = NotFound desc = could not find container \"80f487ffb5dcf6fc6d14099c357fbc7e27472b4a78950699c63c53d90770256a\": container with ID starting with 80f487ffb5dcf6fc6d14099c357fbc7e27472b4a78950699c63c53d90770256a not found: ID does not exist"
Oct 30 21:50:48.746089 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.746030    5678 scope.go:122] "RemoveContainer" containerID="29ceada24258c6d887579990630dc0a842a2ba2a018b98512da626fb7f4b54d9"
Oct 30 21:50:48.746227 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.746211    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29ceada24258c6d887579990630dc0a842a2ba2a018b98512da626fb7f4b54d9"} err="failed to get container status \"29ceada24258c6d887579990630dc0a842a2ba2a018b98512da626fb7f4b54d9\": rpc error: code = NotFound desc = could not find container \"29ceada24258c6d887579990630dc0a842a2ba2a018b98512da626fb7f4b54d9\": container with ID starting with 29ceada24258c6d887579990630dc0a842a2ba2a018b98512da626fb7f4b54d9 not found: ID does not exist"
Oct 30 21:50:48.746273 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.746228    5678 scope.go:122] "RemoveContainer" containerID="258fd66deee3797d90e92c40203fe9ebf6228e43b399cf1d0332b8034deed533"
Oct 30 21:50:48.746509 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.746419    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"258fd66deee3797d90e92c40203fe9ebf6228e43b399cf1d0332b8034deed533"} err="failed to get container status \"258fd66deee3797d90e92c40203fe9ebf6228e43b399cf1d0332b8034deed533\": rpc error: code = NotFound desc = could not find container \"258fd66deee3797d90e92c40203fe9ebf6228e43b399cf1d0332b8034deed533\": container with ID starting with 258fd66deee3797d90e92c40203fe9ebf6228e43b399cf1d0332b8034deed533 not found: ID does not exist"
Oct 30 21:50:48.746509 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.746440    5678 scope.go:122] "RemoveContainer" containerID="9c8be5ed7b18aa2995dfcea5a35d31fa9f057c30716bfa5b89553003d7cc68d2"
Oct 30 21:50:48.746680 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.746660    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c8be5ed7b18aa2995dfcea5a35d31fa9f057c30716bfa5b89553003d7cc68d2"} err="failed to get container status \"9c8be5ed7b18aa2995dfcea5a35d31fa9f057c30716bfa5b89553003d7cc68d2\": rpc error: code = NotFound desc = could not find container \"9c8be5ed7b18aa2995dfcea5a35d31fa9f057c30716bfa5b89553003d7cc68d2\": container with ID starting with 9c8be5ed7b18aa2995dfcea5a35d31fa9f057c30716bfa5b89553003d7cc68d2 not found: ID does not exist"
Oct 30 21:50:48.746680 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.746677    5678 scope.go:122] "RemoveContainer" containerID="1ff12d7276ddc77131dc926b364267852a917f820b7e54191df7b71e90f45ca1"
Oct 30 21:50:48.746903 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.746886    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ff12d7276ddc77131dc926b364267852a917f820b7e54191df7b71e90f45ca1"} err="failed to get container status \"1ff12d7276ddc77131dc926b364267852a917f820b7e54191df7b71e90f45ca1\": rpc error: code = NotFound desc = could not find container \"1ff12d7276ddc77131dc926b364267852a917f820b7e54191df7b71e90f45ca1\": container with ID starting with 1ff12d7276ddc77131dc926b364267852a917f820b7e54191df7b71e90f45ca1 not found: ID does not exist"
Oct 30 21:50:48.746955 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.746904    5678 scope.go:122] "RemoveContainer" containerID="b6e73fa7c904d0a775556c74faaf3620b284efffebf5489198770681155a28d0"
Oct 30 21:50:48.747131 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.747115    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6e73fa7c904d0a775556c74faaf3620b284efffebf5489198770681155a28d0"} err="failed to get container status \"b6e73fa7c904d0a775556c74faaf3620b284efffebf5489198770681155a28d0\": rpc error: code = NotFound desc = could not find container \"b6e73fa7c904d0a775556c74faaf3620b284efffebf5489198770681155a28d0\": container with ID starting with b6e73fa7c904d0a775556c74faaf3620b284efffebf5489198770681155a28d0 not found: ID does not exist"
Oct 30 21:50:48.747190 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.747134    5678 scope.go:122] "RemoveContainer" containerID="7f01cffd7d46651522100fea81bb705ec595c21b7479821b5b352982cd7d4988"
Oct 30 21:50:48.747354 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.747338    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f01cffd7d46651522100fea81bb705ec595c21b7479821b5b352982cd7d4988"} err="failed to get container status \"7f01cffd7d46651522100fea81bb705ec595c21b7479821b5b352982cd7d4988\": rpc error: code = NotFound desc = could not find container \"7f01cffd7d46651522100fea81bb705ec595c21b7479821b5b352982cd7d4988\": container with ID starting with 7f01cffd7d46651522100fea81bb705ec595c21b7479821b5b352982cd7d4988 not found: ID does not exist"
Oct 30 21:50:48.747421 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.747356    5678 scope.go:122] "RemoveContainer" containerID="80f487ffb5dcf6fc6d14099c357fbc7e27472b4a78950699c63c53d90770256a"
Oct 30 21:50:48.747591 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.747574    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80f487ffb5dcf6fc6d14099c357fbc7e27472b4a78950699c63c53d90770256a"} err="failed to get container status \"80f487ffb5dcf6fc6d14099c357fbc7e27472b4a78950699c63c53d90770256a\": rpc error: code = NotFound desc = could not find container \"80f487ffb5dcf6fc6d14099c357fbc7e27472b4a78950699c63c53d90770256a\": container with ID starting with 80f487ffb5dcf6fc6d14099c357fbc7e27472b4a78950699c63c53d90770256a not found: ID does not exist"
Oct 30 21:50:48.747635 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.747591    5678 scope.go:122] "RemoveContainer" containerID="29ceada24258c6d887579990630dc0a842a2ba2a018b98512da626fb7f4b54d9"
Oct 30 21:50:48.747785 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.747766    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29ceada24258c6d887579990630dc0a842a2ba2a018b98512da626fb7f4b54d9"} err="failed to get container status \"29ceada24258c6d887579990630dc0a842a2ba2a018b98512da626fb7f4b54d9\": rpc error: code = NotFound desc = could not find container \"29ceada24258c6d887579990630dc0a842a2ba2a018b98512da626fb7f4b54d9\": container with ID starting with 29ceada24258c6d887579990630dc0a842a2ba2a018b98512da626fb7f4b54d9 not found: ID does not exist"
Oct 30 21:50:48.747852 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.747786    5678 scope.go:122] "RemoveContainer" containerID="258fd66deee3797d90e92c40203fe9ebf6228e43b399cf1d0332b8034deed533"
Oct 30 21:50:48.748085 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.748064    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"258fd66deee3797d90e92c40203fe9ebf6228e43b399cf1d0332b8034deed533"} err="failed to get container status \"258fd66deee3797d90e92c40203fe9ebf6228e43b399cf1d0332b8034deed533\": rpc error: code = NotFound desc = could not find container \"258fd66deee3797d90e92c40203fe9ebf6228e43b399cf1d0332b8034deed533\": container with ID starting with 258fd66deee3797d90e92c40203fe9ebf6228e43b399cf1d0332b8034deed533 not found: ID does not exist"
Oct 30 21:50:48.748085 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.748084    5678 scope.go:122] "RemoveContainer" containerID="9c8be5ed7b18aa2995dfcea5a35d31fa9f057c30716bfa5b89553003d7cc68d2"
Oct 30 21:50:48.748331 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.748315    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c8be5ed7b18aa2995dfcea5a35d31fa9f057c30716bfa5b89553003d7cc68d2"} err="failed to get container status \"9c8be5ed7b18aa2995dfcea5a35d31fa9f057c30716bfa5b89553003d7cc68d2\": rpc error: code = NotFound desc = could not find container \"9c8be5ed7b18aa2995dfcea5a35d31fa9f057c30716bfa5b89553003d7cc68d2\": container with ID starting with 9c8be5ed7b18aa2995dfcea5a35d31fa9f057c30716bfa5b89553003d7cc68d2 not found: ID does not exist"
Oct 30 21:50:48.748331 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.748331    5678 scope.go:122] "RemoveContainer" containerID="1ff12d7276ddc77131dc926b364267852a917f820b7e54191df7b71e90f45ca1"
Oct 30 21:50:48.748552 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.748534    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ff12d7276ddc77131dc926b364267852a917f820b7e54191df7b71e90f45ca1"} err="failed to get container status \"1ff12d7276ddc77131dc926b364267852a917f820b7e54191df7b71e90f45ca1\": rpc error: code = NotFound desc = could not find container \"1ff12d7276ddc77131dc926b364267852a917f820b7e54191df7b71e90f45ca1\": container with ID starting with 1ff12d7276ddc77131dc926b364267852a917f820b7e54191df7b71e90f45ca1 not found: ID does not exist"
Oct 30 21:50:48.748588 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.748553    5678 scope.go:122] "RemoveContainer" containerID="b6e73fa7c904d0a775556c74faaf3620b284efffebf5489198770681155a28d0"
Oct 30 21:50:48.748760 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.748744    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6e73fa7c904d0a775556c74faaf3620b284efffebf5489198770681155a28d0"} err="failed to get container status \"b6e73fa7c904d0a775556c74faaf3620b284efffebf5489198770681155a28d0\": rpc error: code = NotFound desc = could not find container \"b6e73fa7c904d0a775556c74faaf3620b284efffebf5489198770681155a28d0\": container with ID starting with b6e73fa7c904d0a775556c74faaf3620b284efffebf5489198770681155a28d0 not found: ID does not exist"
Oct 30 21:50:48.748817 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.748760    5678 scope.go:122] "RemoveContainer" containerID="7f01cffd7d46651522100fea81bb705ec595c21b7479821b5b352982cd7d4988"
Oct 30 21:50:48.748948 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.748931    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f01cffd7d46651522100fea81bb705ec595c21b7479821b5b352982cd7d4988"} err="failed to get container status \"7f01cffd7d46651522100fea81bb705ec595c21b7479821b5b352982cd7d4988\": rpc error: code = NotFound desc = could not find container \"7f01cffd7d46651522100fea81bb705ec595c21b7479821b5b352982cd7d4988\": container with ID starting with 7f01cffd7d46651522100fea81bb705ec595c21b7479821b5b352982cd7d4988 not found: ID does not exist"
Oct 30 21:50:48.749001 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.748948    5678 scope.go:122] "RemoveContainer" containerID="80f487ffb5dcf6fc6d14099c357fbc7e27472b4a78950699c63c53d90770256a"
Oct 30 21:50:48.749180 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.749163    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80f487ffb5dcf6fc6d14099c357fbc7e27472b4a78950699c63c53d90770256a"} err="failed to get container status \"80f487ffb5dcf6fc6d14099c357fbc7e27472b4a78950699c63c53d90770256a\": rpc error: code = NotFound desc = could not find container \"80f487ffb5dcf6fc6d14099c357fbc7e27472b4a78950699c63c53d90770256a\": container with ID starting with 80f487ffb5dcf6fc6d14099c357fbc7e27472b4a78950699c63c53d90770256a not found: ID does not exist"
Oct 30 21:50:48.749180 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.749180    5678 scope.go:122] "RemoveContainer" containerID="29ceada24258c6d887579990630dc0a842a2ba2a018b98512da626fb7f4b54d9"
Oct 30 21:50:48.749377 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.749362    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29ceada24258c6d887579990630dc0a842a2ba2a018b98512da626fb7f4b54d9"} err="failed to get container status \"29ceada24258c6d887579990630dc0a842a2ba2a018b98512da626fb7f4b54d9\": rpc error: code = NotFound desc = could not find container \"29ceada24258c6d887579990630dc0a842a2ba2a018b98512da626fb7f4b54d9\": container with ID starting with 29ceada24258c6d887579990630dc0a842a2ba2a018b98512da626fb7f4b54d9 not found: ID does not exist"
Oct 30 21:50:48.749434 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.749377    5678 scope.go:122] "RemoveContainer" containerID="258fd66deee3797d90e92c40203fe9ebf6228e43b399cf1d0332b8034deed533"
Oct 30 21:50:48.749607 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.749587    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"258fd66deee3797d90e92c40203fe9ebf6228e43b399cf1d0332b8034deed533"} err="failed to get container status \"258fd66deee3797d90e92c40203fe9ebf6228e43b399cf1d0332b8034deed533\": rpc error: code = NotFound desc = could not find container \"258fd66deee3797d90e92c40203fe9ebf6228e43b399cf1d0332b8034deed533\": container with ID starting with 258fd66deee3797d90e92c40203fe9ebf6228e43b399cf1d0332b8034deed533 not found: ID does not exist"
Oct 30 21:50:48.816901 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.816871    5678 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-web-config\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:50:48.816901 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.816901    5678 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-thanos-prometheus-http-client-file\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:50:48.817099 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.816916    5678 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:50:48.817099 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.816932    5678 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-tls-assets\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:50:48.817099 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.816947    5678 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-config-out\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:50:48.817099 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.816960    5678 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-config\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:50:48.817099 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.816975    5678 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:50:48.817099 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.816990    5678 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:50:48.817099 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.817005    5678 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:50:48.817099 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.817022    5678 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:50:48.817099 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.817037    5678 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-secret-prometheus-k8s-tls\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:50:48.817099 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.817052    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cr8x2\" (UniqueName: \"kubernetes.io/projected/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-kube-api-access-cr8x2\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:50:48.817099 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.817067    5678 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-secret-metrics-client-certs\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:50:48.817099 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.817082    5678 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-prometheus-trusted-ca-bundle\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:50:48.817099 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.817099    5678 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-configmap-metrics-client-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:50:48.817533 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.817115    5678 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-secret-grpc-tls\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:50:48.817533 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.817130    5678 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-secret-kube-rbac-proxy\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:50:48.817533 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.817146    5678 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef7ae922-6da7-4817-ba91-35e67b6ac8c5-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:50:48.887580 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:48.887548    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7cbdbcf4b5-rmbsz"]
Oct 30 21:50:49.119106 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:49.119072    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 21:50:49.124197 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:49.124174    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 21:50:49.170438 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:49.170356    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 21:50:49.170621 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:49.170608    5678 cpu_manager.go:414] "containerMap: removing container" podUID="ef7ae922-6da7-4817-ba91-35e67b6ac8c5" containerName="kube-rbac-proxy-thanos"
Oct 30 21:50:49.170677 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:49.170622    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef7ae922-6da7-4817-ba91-35e67b6ac8c5" containerName="kube-rbac-proxy-thanos"
Oct 30 21:50:49.170677 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:49.170634    5678 cpu_manager.go:414] "containerMap: removing container" podUID="5138fe8c-9927-4d05-8fe5-5556072ed595" containerName="config-reloader"
Oct 30 21:50:49.170677 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:49.170639    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5138fe8c-9927-4d05-8fe5-5556072ed595" containerName="config-reloader"
Oct 30 21:50:49.170677 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:49.170646    5678 cpu_manager.go:414] "containerMap: removing container" podUID="ef7ae922-6da7-4817-ba91-35e67b6ac8c5" containerName="prometheus"
Oct 30 21:50:49.170677 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:49.170650    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef7ae922-6da7-4817-ba91-35e67b6ac8c5" containerName="prometheus"
Oct 30 21:50:49.170677 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:49.170665    5678 cpu_manager.go:414] "containerMap: removing container" podUID="5138fe8c-9927-4d05-8fe5-5556072ed595" containerName="kube-rbac-proxy-web"
Oct 30 21:50:49.170677 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:49.170672    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5138fe8c-9927-4d05-8fe5-5556072ed595" containerName="kube-rbac-proxy-web"
Oct 30 21:50:49.170677 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:49.170680    5678 cpu_manager.go:414] "containerMap: removing container" podUID="5138fe8c-9927-4d05-8fe5-5556072ed595" containerName="kube-rbac-proxy"
Oct 30 21:50:49.171209 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:49.170687    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5138fe8c-9927-4d05-8fe5-5556072ed595" containerName="kube-rbac-proxy"
Oct 30 21:50:49.171209 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:49.170697    5678 cpu_manager.go:414] "containerMap: removing container" podUID="5138fe8c-9927-4d05-8fe5-5556072ed595" containerName="init-config-reloader"
Oct 30 21:50:49.171209 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:49.170704    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5138fe8c-9927-4d05-8fe5-5556072ed595" containerName="init-config-reloader"
Oct 30 21:50:49.171209 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:49.170718    5678 cpu_manager.go:414] "containerMap: removing container" podUID="5138fe8c-9927-4d05-8fe5-5556072ed595" containerName="prom-label-proxy"
Oct 30 21:50:49.171209 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:49.170725    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5138fe8c-9927-4d05-8fe5-5556072ed595" containerName="prom-label-proxy"
Oct 30 21:50:49.171209 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:49.170737    5678 cpu_manager.go:414] "containerMap: removing container" podUID="ef7ae922-6da7-4817-ba91-35e67b6ac8c5" containerName="config-reloader"
Oct 30 21:50:49.171209 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:49.170744    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef7ae922-6da7-4817-ba91-35e67b6ac8c5" containerName="config-reloader"
Oct 30 21:50:49.171209 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:49.170756    5678 cpu_manager.go:414] "containerMap: removing container" podUID="5138fe8c-9927-4d05-8fe5-5556072ed595" containerName="kube-rbac-proxy-metric"
Oct 30 21:50:49.171209 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:49.170763    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5138fe8c-9927-4d05-8fe5-5556072ed595" containerName="kube-rbac-proxy-metric"
Oct 30 21:50:49.171209 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:49.170777    5678 cpu_manager.go:414] "containerMap: removing container" podUID="ef7ae922-6da7-4817-ba91-35e67b6ac8c5" containerName="thanos-sidecar"
Oct 30 21:50:49.171209 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:49.170784    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef7ae922-6da7-4817-ba91-35e67b6ac8c5" containerName="thanos-sidecar"
Oct 30 21:50:49.171209 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:49.170793    5678 cpu_manager.go:414] "containerMap: removing container" podUID="ef7ae922-6da7-4817-ba91-35e67b6ac8c5" containerName="kube-rbac-proxy"
Oct 30 21:50:49.171209 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:49.170800    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef7ae922-6da7-4817-ba91-35e67b6ac8c5" containerName="kube-rbac-proxy"
Oct 30 21:50:49.171209 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:49.170810    5678 cpu_manager.go:414] "containerMap: removing container" podUID="5138fe8c-9927-4d05-8fe5-5556072ed595" containerName="alertmanager"
Oct 30 21:50:49.171209 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:49.170815    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5138fe8c-9927-4d05-8fe5-5556072ed595" containerName="alertmanager"
Oct 30 21:50:49.171209 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:49.170821    5678 cpu_manager.go:414] "containerMap: removing container" podUID="ef7ae922-6da7-4817-ba91-35e67b6ac8c5" containerName="init-config-reloader"
Oct 30 21:50:49.171209 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:49.170827    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef7ae922-6da7-4817-ba91-35e67b6ac8c5" containerName="init-config-reloader"
Oct 30 21:50:49.171209 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:49.170833    5678 cpu_manager.go:414] "containerMap: removing container" podUID="ef7ae922-6da7-4817-ba91-35e67b6ac8c5" containerName="kube-rbac-proxy-web"
Oct 30 21:50:49.171209 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:49.170837    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef7ae922-6da7-4817-ba91-35e67b6ac8c5" containerName="kube-rbac-proxy-web"
Oct 30 21:50:49.171209 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:49.170914    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="5138fe8c-9927-4d05-8fe5-5556072ed595" containerName="kube-rbac-proxy-web"
Oct 30 21:50:49.171209 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:49.170927    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="5138fe8c-9927-4d05-8fe5-5556072ed595" containerName="kube-rbac-proxy"
Oct 30 21:50:49.171209 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:49.170938    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="5138fe8c-9927-4d05-8fe5-5556072ed595" containerName="init-config-reloader"
Oct 30 21:50:49.171209 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:49.170948    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="5138fe8c-9927-4d05-8fe5-5556072ed595" containerName="prom-label-proxy"
Oct 30 21:50:49.171209 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:49.170957    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef7ae922-6da7-4817-ba91-35e67b6ac8c5" containerName="config-reloader"
Oct 30 21:50:49.171209 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:49.170966    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="5138fe8c-9927-4d05-8fe5-5556072ed595" containerName="kube-rbac-proxy-metric"
Oct 30 21:50:49.171209 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:49.170975    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef7ae922-6da7-4817-ba91-35e67b6ac8c5" containerName="thanos-sidecar"
Oct 30 21:50:49.171209 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:49.170984    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef7ae922-6da7-4817-ba91-35e67b6ac8c5" containerName="kube-rbac-proxy"
Oct 30 21:50:49.171209 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:49.170994    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="5138fe8c-9927-4d05-8fe5-5556072ed595" containerName="alertmanager"
Oct 30 21:50:49.171209 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:49.171002    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef7ae922-6da7-4817-ba91-35e67b6ac8c5" containerName="kube-rbac-proxy-web"
Oct 30 21:50:49.171209 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:49.171012    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef7ae922-6da7-4817-ba91-35e67b6ac8c5" containerName="kube-rbac-proxy-thanos"
Oct 30 21:50:49.171209 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:49.171020    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="5138fe8c-9927-4d05-8fe5-5556072ed595" containerName="config-reloader"
Oct 30 21:50:49.171209 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:49.171029    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef7ae922-6da7-4817-ba91-35e67b6ac8c5" containerName="prometheus"
Oct 30 21:50:49.175035 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:49.175019    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:49.227912 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:49.227887    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Oct 30 21:50:49.228223 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:49.228207    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Oct 30 21:50:49.228431 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:49.228394    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Oct 30 21:50:49.275926 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:49.275896    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Oct 30 21:50:49.275926 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:49.275901    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Oct 30 21:50:49.276146 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:49.275947    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Oct 30 21:50:49.276146 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:49.275968    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Oct 30 21:50:49.276146 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:49.275968    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Oct 30 21:50:49.276146 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:49.275996    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Oct 30 21:50:49.276146 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:49.276012    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Oct 30 21:50:49.276146 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:49.276033    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Oct 30 21:50:49.276146 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:49.276075    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-kfpck\""
Oct 30 21:50:49.276146 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:49.276131    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Oct 30 21:50:49.276441 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:49.276175    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Oct 30 21:50:49.276441 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:49.276252    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Oct 30 21:50:49.276441 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:49.276309    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-bmu48km85ucup\""
Oct 30 21:50:49.277044 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:49.277028    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 21:50:49.320733 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:49.320711    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/11d752aa-0037-43c4-aadc-1bd78e14918e-config\") pod \"prometheus-k8s-0\" (UID: \"11d752aa-0037-43c4-aadc-1bd78e14918e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:49.320863 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:49.320738    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/11d752aa-0037-43c4-aadc-1bd78e14918e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"11d752aa-0037-43c4-aadc-1bd78e14918e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:49.320863 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:49.320765    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f589t\" (UniqueName: \"kubernetes.io/projected/11d752aa-0037-43c4-aadc-1bd78e14918e-kube-api-access-f589t\") pod \"prometheus-k8s-0\" (UID: \"11d752aa-0037-43c4-aadc-1bd78e14918e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:49.320863 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:49.320782    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/11d752aa-0037-43c4-aadc-1bd78e14918e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"11d752aa-0037-43c4-aadc-1bd78e14918e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:49.321042 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:49.320922    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/11d752aa-0037-43c4-aadc-1bd78e14918e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"11d752aa-0037-43c4-aadc-1bd78e14918e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:49.321042 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:49.320959    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/11d752aa-0037-43c4-aadc-1bd78e14918e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"11d752aa-0037-43c4-aadc-1bd78e14918e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:49.321042 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:49.321007    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11d752aa-0037-43c4-aadc-1bd78e14918e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"11d752aa-0037-43c4-aadc-1bd78e14918e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:49.321042 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:49.321037    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/11d752aa-0037-43c4-aadc-1bd78e14918e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"11d752aa-0037-43c4-aadc-1bd78e14918e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:49.321186 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:49.321062    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/11d752aa-0037-43c4-aadc-1bd78e14918e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"11d752aa-0037-43c4-aadc-1bd78e14918e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:49.321186 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:49.321087    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11d752aa-0037-43c4-aadc-1bd78e14918e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"11d752aa-0037-43c4-aadc-1bd78e14918e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:49.321186 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:49.321155    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/11d752aa-0037-43c4-aadc-1bd78e14918e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"11d752aa-0037-43c4-aadc-1bd78e14918e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:49.321186 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:49.321175    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/11d752aa-0037-43c4-aadc-1bd78e14918e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"11d752aa-0037-43c4-aadc-1bd78e14918e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:49.321323 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:49.321195    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/11d752aa-0037-43c4-aadc-1bd78e14918e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"11d752aa-0037-43c4-aadc-1bd78e14918e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:49.321323 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:49.321212    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/11d752aa-0037-43c4-aadc-1bd78e14918e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"11d752aa-0037-43c4-aadc-1bd78e14918e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:49.321323 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:49.321287    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11d752aa-0037-43c4-aadc-1bd78e14918e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"11d752aa-0037-43c4-aadc-1bd78e14918e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:49.321323 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:49.321319    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/11d752aa-0037-43c4-aadc-1bd78e14918e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"11d752aa-0037-43c4-aadc-1bd78e14918e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:49.321525 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:49.321348    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/11d752aa-0037-43c4-aadc-1bd78e14918e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"11d752aa-0037-43c4-aadc-1bd78e14918e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:49.321525 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:49.321363    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/11d752aa-0037-43c4-aadc-1bd78e14918e-web-config\") pod \"prometheus-k8s-0\" (UID: \"11d752aa-0037-43c4-aadc-1bd78e14918e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:49.321525 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:49.321378    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/11d752aa-0037-43c4-aadc-1bd78e14918e-config-out\") pod \"prometheus-k8s-0\" (UID: \"11d752aa-0037-43c4-aadc-1bd78e14918e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:49.321525 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:49.321438    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/11d752aa-0037-43c4-aadc-1bd78e14918e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"11d752aa-0037-43c4-aadc-1bd78e14918e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:49.422091 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:49.422015    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/11d752aa-0037-43c4-aadc-1bd78e14918e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"11d752aa-0037-43c4-aadc-1bd78e14918e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:49.422091 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:49.422051    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/11d752aa-0037-43c4-aadc-1bd78e14918e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"11d752aa-0037-43c4-aadc-1bd78e14918e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:49.422091 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:49.422070    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11d752aa-0037-43c4-aadc-1bd78e14918e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"11d752aa-0037-43c4-aadc-1bd78e14918e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:49.422091 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:49.422088    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/11d752aa-0037-43c4-aadc-1bd78e14918e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"11d752aa-0037-43c4-aadc-1bd78e14918e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:49.422375 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:49.422104    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/11d752aa-0037-43c4-aadc-1bd78e14918e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"11d752aa-0037-43c4-aadc-1bd78e14918e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:49.422375 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:49.422118    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11d752aa-0037-43c4-aadc-1bd78e14918e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"11d752aa-0037-43c4-aadc-1bd78e14918e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:49.422375 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:49.422138    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/11d752aa-0037-43c4-aadc-1bd78e14918e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"11d752aa-0037-43c4-aadc-1bd78e14918e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:49.422375 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:49.422152    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/11d752aa-0037-43c4-aadc-1bd78e14918e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"11d752aa-0037-43c4-aadc-1bd78e14918e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:49.422375 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:49.422172    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/11d752aa-0037-43c4-aadc-1bd78e14918e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"11d752aa-0037-43c4-aadc-1bd78e14918e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:49.422375 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:49.422188    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/11d752aa-0037-43c4-aadc-1bd78e14918e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"11d752aa-0037-43c4-aadc-1bd78e14918e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:49.422375 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:49.422221    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11d752aa-0037-43c4-aadc-1bd78e14918e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"11d752aa-0037-43c4-aadc-1bd78e14918e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:49.422375 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:49.422248    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/11d752aa-0037-43c4-aadc-1bd78e14918e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"11d752aa-0037-43c4-aadc-1bd78e14918e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:49.422375 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:49.422278    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/11d752aa-0037-43c4-aadc-1bd78e14918e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"11d752aa-0037-43c4-aadc-1bd78e14918e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:49.422375 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:49.422300    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/11d752aa-0037-43c4-aadc-1bd78e14918e-web-config\") pod \"prometheus-k8s-0\" (UID: \"11d752aa-0037-43c4-aadc-1bd78e14918e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:49.422375 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:49.422326    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/11d752aa-0037-43c4-aadc-1bd78e14918e-config-out\") pod \"prometheus-k8s-0\" (UID: \"11d752aa-0037-43c4-aadc-1bd78e14918e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:49.422375 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:49.422352    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/11d752aa-0037-43c4-aadc-1bd78e14918e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"11d752aa-0037-43c4-aadc-1bd78e14918e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:49.422993 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:49.422393    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/11d752aa-0037-43c4-aadc-1bd78e14918e-config\") pod \"prometheus-k8s-0\" (UID: \"11d752aa-0037-43c4-aadc-1bd78e14918e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:49.422993 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:49.422443    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/11d752aa-0037-43c4-aadc-1bd78e14918e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"11d752aa-0037-43c4-aadc-1bd78e14918e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:49.422993 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:49.422481    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f589t\" (UniqueName: \"kubernetes.io/projected/11d752aa-0037-43c4-aadc-1bd78e14918e-kube-api-access-f589t\") pod \"prometheus-k8s-0\" (UID: \"11d752aa-0037-43c4-aadc-1bd78e14918e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:49.422993 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:49.422508    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/11d752aa-0037-43c4-aadc-1bd78e14918e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"11d752aa-0037-43c4-aadc-1bd78e14918e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:49.422993 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:49.422528    5678 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 21:50:49.422993 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:49.422919    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11d752aa-0037-43c4-aadc-1bd78e14918e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"11d752aa-0037-43c4-aadc-1bd78e14918e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:49.422993 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:49.422967    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/11d752aa-0037-43c4-aadc-1bd78e14918e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"11d752aa-0037-43c4-aadc-1bd78e14918e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:49.423381 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:49.423326    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11d752aa-0037-43c4-aadc-1bd78e14918e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"11d752aa-0037-43c4-aadc-1bd78e14918e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:49.424162 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:49.424135    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/11d752aa-0037-43c4-aadc-1bd78e14918e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"11d752aa-0037-43c4-aadc-1bd78e14918e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:49.425544 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:49.425519    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/11d752aa-0037-43c4-aadc-1bd78e14918e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"11d752aa-0037-43c4-aadc-1bd78e14918e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:49.425758 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:49.425682    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/11d752aa-0037-43c4-aadc-1bd78e14918e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"11d752aa-0037-43c4-aadc-1bd78e14918e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:49.425758 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:49.425715    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/11d752aa-0037-43c4-aadc-1bd78e14918e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"11d752aa-0037-43c4-aadc-1bd78e14918e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:49.425895 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:49.425761    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/11d752aa-0037-43c4-aadc-1bd78e14918e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"11d752aa-0037-43c4-aadc-1bd78e14918e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:49.426213 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:49.426032    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/11d752aa-0037-43c4-aadc-1bd78e14918e-config\") pod \"prometheus-k8s-0\" (UID: \"11d752aa-0037-43c4-aadc-1bd78e14918e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:49.426213 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:49.426159    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/11d752aa-0037-43c4-aadc-1bd78e14918e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"11d752aa-0037-43c4-aadc-1bd78e14918e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:49.426527 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:49.426483    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11d752aa-0037-43c4-aadc-1bd78e14918e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"11d752aa-0037-43c4-aadc-1bd78e14918e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:49.426640 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:49.426541    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/11d752aa-0037-43c4-aadc-1bd78e14918e-config-out\") pod \"prometheus-k8s-0\" (UID: \"11d752aa-0037-43c4-aadc-1bd78e14918e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:49.426698 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:49.426648    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/11d752aa-0037-43c4-aadc-1bd78e14918e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"11d752aa-0037-43c4-aadc-1bd78e14918e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:49.426865 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:49.426820    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/11d752aa-0037-43c4-aadc-1bd78e14918e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"11d752aa-0037-43c4-aadc-1bd78e14918e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:49.427206 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:49.427182    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/11d752aa-0037-43c4-aadc-1bd78e14918e-web-config\") pod \"prometheus-k8s-0\" (UID: \"11d752aa-0037-43c4-aadc-1bd78e14918e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:49.428080 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:49.428057    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/11d752aa-0037-43c4-aadc-1bd78e14918e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"11d752aa-0037-43c4-aadc-1bd78e14918e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:49.428305 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:49.428284    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/11d752aa-0037-43c4-aadc-1bd78e14918e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"11d752aa-0037-43c4-aadc-1bd78e14918e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:49.428603 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:49.428584    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/11d752aa-0037-43c4-aadc-1bd78e14918e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"11d752aa-0037-43c4-aadc-1bd78e14918e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:49.428798 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:49.428779    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/11d752aa-0037-43c4-aadc-1bd78e14918e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"11d752aa-0037-43c4-aadc-1bd78e14918e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:49.488945 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:49.488920    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f589t\" (UniqueName: \"kubernetes.io/projected/11d752aa-0037-43c4-aadc-1bd78e14918e-kube-api-access-f589t\") pod \"prometheus-k8s-0\" (UID: \"11d752aa-0037-43c4-aadc-1bd78e14918e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:49.492831 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:49.492810    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:50:49.915528 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:49.915499    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 21:50:49.918525 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:50:49.918496    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11d752aa_0037_43c4_aadc_1bd78e14918e.slice/crio-91c15b39e48b5d43a717792d6ee3b6db180225120c0d7a38cb143684fb704c3c WatchSource:0}: Error finding container 91c15b39e48b5d43a717792d6ee3b6db180225120c0d7a38cb143684fb704c3c: Status 404 returned error can't find the container with id 91c15b39e48b5d43a717792d6ee3b6db180225120c0d7a38cb143684fb704c3c
Oct 30 21:50:50.268573 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:50.268490    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-w9tx8"
Oct 30 21:50:50.519841 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:50.519773    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef7ae922-6da7-4817-ba91-35e67b6ac8c5" path="/var/lib/kubelet/pods/ef7ae922-6da7-4817-ba91-35e67b6ac8c5/volumes"
Oct 30 21:50:50.619357 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:50.619319    5678 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-69874d4cdd-zhlp5" podUID="69e218ec-ab5e-4e09-8444-e53a710c53bf" containerName="console" containerID="cri-o://510dece2cd11b7771e421f49c46269e9e21714206ddf570252332975593296c6" gracePeriod=15
Oct 30 21:50:50.700265 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:50.700238    5678 generic.go:356] "Generic (PLEG): container finished" podID="11d752aa-0037-43c4-aadc-1bd78e14918e" containerID="ae5631caf6aa81f7290ebb2474b55bb6e90e3303646f2f68c4ebe946055429df" exitCode=0
Oct 30 21:50:50.700411 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:50.700328    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"11d752aa-0037-43c4-aadc-1bd78e14918e","Type":"ContainerDied","Data":"ae5631caf6aa81f7290ebb2474b55bb6e90e3303646f2f68c4ebe946055429df"}
Oct 30 21:50:50.700468 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:50.700410    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"11d752aa-0037-43c4-aadc-1bd78e14918e","Type":"ContainerStarted","Data":"91c15b39e48b5d43a717792d6ee3b6db180225120c0d7a38cb143684fb704c3c"}
Oct 30 21:50:50.905191 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:50.905024    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69874d4cdd-zhlp5"
Oct 30 21:50:51.032494 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:51.032469    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69e218ec-ab5e-4e09-8444-e53a710c53bf-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/69e218ec-ab5e-4e09-8444-e53a710c53bf-oauth-serving-cert\") pod \"69e218ec-ab5e-4e09-8444-e53a710c53bf\" (UID: \"69e218ec-ab5e-4e09-8444-e53a710c53bf\") "
Oct 30 21:50:51.033007 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:51.032504    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/69e218ec-ab5e-4e09-8444-e53a710c53bf-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/69e218ec-ab5e-4e09-8444-e53a710c53bf-console-serving-cert\") pod \"69e218ec-ab5e-4e09-8444-e53a710c53bf\" (UID: \"69e218ec-ab5e-4e09-8444-e53a710c53bf\") "
Oct 30 21:50:51.033007 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:51.032524    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69e218ec-ab5e-4e09-8444-e53a710c53bf-console-config\" (UniqueName: \"kubernetes.io/configmap/69e218ec-ab5e-4e09-8444-e53a710c53bf-console-config\") pod \"69e218ec-ab5e-4e09-8444-e53a710c53bf\" (UID: \"69e218ec-ab5e-4e09-8444-e53a710c53bf\") "
Oct 30 21:50:51.033007 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:51.032554    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69e218ec-ab5e-4e09-8444-e53a710c53bf-tmp\" (UniqueName: \"kubernetes.io/empty-dir/69e218ec-ab5e-4e09-8444-e53a710c53bf-tmp\") pod \"69e218ec-ab5e-4e09-8444-e53a710c53bf\" (UID: \"69e218ec-ab5e-4e09-8444-e53a710c53bf\") "
Oct 30 21:50:51.033007 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:51.032593    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69e218ec-ab5e-4e09-8444-e53a710c53bf-kube-api-access-gv6wf\" (UniqueName: \"kubernetes.io/projected/69e218ec-ab5e-4e09-8444-e53a710c53bf-kube-api-access-gv6wf\") pod \"69e218ec-ab5e-4e09-8444-e53a710c53bf\" (UID: \"69e218ec-ab5e-4e09-8444-e53a710c53bf\") "
Oct 30 21:50:51.033007 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:51.032625    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69e218ec-ab5e-4e09-8444-e53a710c53bf-service-ca\" (UniqueName: \"kubernetes.io/configmap/69e218ec-ab5e-4e09-8444-e53a710c53bf-service-ca\") pod \"69e218ec-ab5e-4e09-8444-e53a710c53bf\" (UID: \"69e218ec-ab5e-4e09-8444-e53a710c53bf\") "
Oct 30 21:50:51.033007 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:51.032669    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/69e218ec-ab5e-4e09-8444-e53a710c53bf-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/69e218ec-ab5e-4e09-8444-e53a710c53bf-console-oauth-config\") pod \"69e218ec-ab5e-4e09-8444-e53a710c53bf\" (UID: \"69e218ec-ab5e-4e09-8444-e53a710c53bf\") "
Oct 30 21:50:51.033007 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:51.032876    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69e218ec-ab5e-4e09-8444-e53a710c53bf-tmp" pod "69e218ec-ab5e-4e09-8444-e53a710c53bf" (UID: "69e218ec-ab5e-4e09-8444-e53a710c53bf"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:50:51.033546 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:51.033122    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69e218ec-ab5e-4e09-8444-e53a710c53bf-oauth-serving-cert" pod "69e218ec-ab5e-4e09-8444-e53a710c53bf" (UID: "69e218ec-ab5e-4e09-8444-e53a710c53bf"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:50:51.033546 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:51.033311    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69e218ec-ab5e-4e09-8444-e53a710c53bf-service-ca" pod "69e218ec-ab5e-4e09-8444-e53a710c53bf" (UID: "69e218ec-ab5e-4e09-8444-e53a710c53bf"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:50:51.033546 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:51.033518    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69e218ec-ab5e-4e09-8444-e53a710c53bf-console-config" pod "69e218ec-ab5e-4e09-8444-e53a710c53bf" (UID: "69e218ec-ab5e-4e09-8444-e53a710c53bf"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:50:51.034999 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:51.034974    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69e218ec-ab5e-4e09-8444-e53a710c53bf-console-serving-cert" pod "69e218ec-ab5e-4e09-8444-e53a710c53bf" (UID: "69e218ec-ab5e-4e09-8444-e53a710c53bf"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:50:51.036054 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:51.036023    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69e218ec-ab5e-4e09-8444-e53a710c53bf-kube-api-access-gv6wf" pod "69e218ec-ab5e-4e09-8444-e53a710c53bf" (UID: "69e218ec-ab5e-4e09-8444-e53a710c53bf"). InnerVolumeSpecName "kube-api-access-gv6wf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:50:51.036212 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:51.036192    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69e218ec-ab5e-4e09-8444-e53a710c53bf-console-oauth-config" pod "69e218ec-ab5e-4e09-8444-e53a710c53bf" (UID: "69e218ec-ab5e-4e09-8444-e53a710c53bf"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:50:51.134043 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:51.133805    5678 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/69e218ec-ab5e-4e09-8444-e53a710c53bf-tmp\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:50:51.134043 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:51.133840    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gv6wf\" (UniqueName: \"kubernetes.io/projected/69e218ec-ab5e-4e09-8444-e53a710c53bf-kube-api-access-gv6wf\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:50:51.134043 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:51.133855    5678 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/69e218ec-ab5e-4e09-8444-e53a710c53bf-service-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:50:51.134043 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:51.133871    5678 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/69e218ec-ab5e-4e09-8444-e53a710c53bf-console-oauth-config\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:50:51.134043 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:51.133885    5678 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/69e218ec-ab5e-4e09-8444-e53a710c53bf-oauth-serving-cert\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:50:51.134043 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:51.133898    5678 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/69e218ec-ab5e-4e09-8444-e53a710c53bf-console-serving-cert\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:50:51.134043 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:51.133912    5678 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/69e218ec-ab5e-4e09-8444-e53a710c53bf-console-config\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:50:51.707167 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:51.707127    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"11d752aa-0037-43c4-aadc-1bd78e14918e","Type":"ContainerStarted","Data":"777dbe7e9ff543b7fa96b816366010e429859de68131c721545b66331ccdef02"}
Oct 30 21:50:51.707342 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:51.707202    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"11d752aa-0037-43c4-aadc-1bd78e14918e","Type":"ContainerStarted","Data":"a640454f9df35697da3ca142cb5c4c0d42467112c8d0ced637224e5a75b31263"}
Oct 30 21:50:51.707342 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:51.707244    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"11d752aa-0037-43c4-aadc-1bd78e14918e","Type":"ContainerStarted","Data":"df0a1a2196155728025f374d6468489eca541348a8d4e9c55c26c178538f65a3"}
Oct 30 21:50:51.707342 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:51.707272    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"11d752aa-0037-43c4-aadc-1bd78e14918e","Type":"ContainerStarted","Data":"dbefc7b0e72374d6b141e0c379cf527cacfed570fa5a9f11d688b113e6a5bc3e"}
Oct 30 21:50:51.707342 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:51.707299    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"11d752aa-0037-43c4-aadc-1bd78e14918e","Type":"ContainerStarted","Data":"7ad381db92343dfe2b341331affe7460984e566c7e4c2d3179217f888965627c"}
Oct 30 21:50:51.707342 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:51.707330    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"11d752aa-0037-43c4-aadc-1bd78e14918e","Type":"ContainerStarted","Data":"4b8bc36754099815421cd9eaa9635c0ffe231f96d12e9b50d039af20b6c4ce0a"}
Oct 30 21:50:51.708331 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:51.708309    5678 generic.go:356] "Generic (PLEG): container finished" podID="69e218ec-ab5e-4e09-8444-e53a710c53bf" containerID="510dece2cd11b7771e421f49c46269e9e21714206ddf570252332975593296c6" exitCode=0
Oct 30 21:50:51.708433 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:51.708356    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69874d4cdd-zhlp5" event={"ID":"69e218ec-ab5e-4e09-8444-e53a710c53bf","Type":"ContainerDied","Data":"510dece2cd11b7771e421f49c46269e9e21714206ddf570252332975593296c6"}
Oct 30 21:50:51.708433 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:51.708362    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69874d4cdd-zhlp5"
Oct 30 21:50:51.708433 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:51.708385    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69874d4cdd-zhlp5" event={"ID":"69e218ec-ab5e-4e09-8444-e53a710c53bf","Type":"ContainerDied","Data":"ca41b3353f4cff5c25e5091dd9731c19f4a1577f6aceaf86e9b91db22ba7f5e1"}
Oct 30 21:50:51.708433 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:51.708421    5678 scope.go:122] "RemoveContainer" containerID="510dece2cd11b7771e421f49c46269e9e21714206ddf570252332975593296c6"
Oct 30 21:50:51.716313 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:51.716292    5678 scope.go:122] "RemoveContainer" containerID="510dece2cd11b7771e421f49c46269e9e21714206ddf570252332975593296c6"
Oct 30 21:50:51.716622 ip-10-0-128-245 kubenswrapper[5678]: E1030 21:50:51.716604    5678 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"510dece2cd11b7771e421f49c46269e9e21714206ddf570252332975593296c6\": container with ID starting with 510dece2cd11b7771e421f49c46269e9e21714206ddf570252332975593296c6 not found: ID does not exist" containerID="510dece2cd11b7771e421f49c46269e9e21714206ddf570252332975593296c6"
Oct 30 21:50:51.716691 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:51.716634    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"510dece2cd11b7771e421f49c46269e9e21714206ddf570252332975593296c6"} err="failed to get container status \"510dece2cd11b7771e421f49c46269e9e21714206ddf570252332975593296c6\": rpc error: code = NotFound desc = could not find container \"510dece2cd11b7771e421f49c46269e9e21714206ddf570252332975593296c6\": container with ID starting with 510dece2cd11b7771e421f49c46269e9e21714206ddf570252332975593296c6 not found: ID does not exist"
Oct 30 21:50:51.834563 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:51.834509    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.818533668 podStartE2EDuration="2.834490971s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.957303ms" isStatefulPod=false podCreationTimestamp="2026-10-30 21:50:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:50:51.833782581 +0000 UTC m=+29.438090131" watchObservedRunningTime="2026-10-30 21:50:51.834490971 +0000 UTC m=+29.438798525"
Oct 30 21:50:51.939052 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:51.939022    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-69874d4cdd-zhlp5"]
Oct 30 21:50:51.944304 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:51.944278    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-69874d4cdd-zhlp5"]
Oct 30 21:50:52.519987 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:52.519956    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69e218ec-ab5e-4e09-8444-e53a710c53bf" path="/var/lib/kubelet/pods/69e218ec-ab5e-4e09-8444-e53a710c53bf/volumes"
Oct 30 21:50:54.493888 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:50:54.493846    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:51:10.569249 ip-10-0-128-245 kubenswrapper[5678]: E1030 21:51:10.569213    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 21:51:13.915106 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:51:13.915066    5678 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-7cbdbcf4b5-rmbsz" podUID="fdf54bb9-f735-4810-82ee-d803099a1bcf" containerName="console" containerID="cri-o://e39713154e1dc737f4cfcb4c40f6f2c41e5c329785e99738a71e93998afec246" gracePeriod=15
Oct 30 21:51:14.162174 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:51:14.162150    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7cbdbcf4b5-rmbsz"
Oct 30 21:51:14.215512 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:51:14.215449    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdf54bb9-f735-4810-82ee-d803099a1bcf-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdf54bb9-f735-4810-82ee-d803099a1bcf-service-ca\") pod \"fdf54bb9-f735-4810-82ee-d803099a1bcf\" (UID: \"fdf54bb9-f735-4810-82ee-d803099a1bcf\") "
Oct 30 21:51:14.215512 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:51:14.215491    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fdf54bb9-f735-4810-82ee-d803099a1bcf-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fdf54bb9-f735-4810-82ee-d803099a1bcf-console-serving-cert\") pod \"fdf54bb9-f735-4810-82ee-d803099a1bcf\" (UID: \"fdf54bb9-f735-4810-82ee-d803099a1bcf\") "
Oct 30 21:51:14.215701 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:51:14.215516    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdf54bb9-f735-4810-82ee-d803099a1bcf-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fdf54bb9-f735-4810-82ee-d803099a1bcf-trusted-ca-bundle\") pod \"fdf54bb9-f735-4810-82ee-d803099a1bcf\" (UID: \"fdf54bb9-f735-4810-82ee-d803099a1bcf\") "
Oct 30 21:51:14.215701 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:51:14.215539    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdf54bb9-f735-4810-82ee-d803099a1bcf-console-config\" (UniqueName: \"kubernetes.io/configmap/fdf54bb9-f735-4810-82ee-d803099a1bcf-console-config\") pod \"fdf54bb9-f735-4810-82ee-d803099a1bcf\" (UID: \"fdf54bb9-f735-4810-82ee-d803099a1bcf\") "
Oct 30 21:51:14.215701 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:51:14.215573    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdf54bb9-f735-4810-82ee-d803099a1bcf-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fdf54bb9-f735-4810-82ee-d803099a1bcf-oauth-serving-cert\") pod \"fdf54bb9-f735-4810-82ee-d803099a1bcf\" (UID: \"fdf54bb9-f735-4810-82ee-d803099a1bcf\") "
Oct 30 21:51:14.215701 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:51:14.215604    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdf54bb9-f735-4810-82ee-d803099a1bcf-kube-api-access-s5jdv\" (UniqueName: \"kubernetes.io/projected/fdf54bb9-f735-4810-82ee-d803099a1bcf-kube-api-access-s5jdv\") pod \"fdf54bb9-f735-4810-82ee-d803099a1bcf\" (UID: \"fdf54bb9-f735-4810-82ee-d803099a1bcf\") "
Oct 30 21:51:14.215701 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:51:14.215686    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdf54bb9-f735-4810-82ee-d803099a1bcf-tmp\" (UniqueName: \"kubernetes.io/empty-dir/fdf54bb9-f735-4810-82ee-d803099a1bcf-tmp\") pod \"fdf54bb9-f735-4810-82ee-d803099a1bcf\" (UID: \"fdf54bb9-f735-4810-82ee-d803099a1bcf\") "
Oct 30 21:51:14.215954 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:51:14.215713    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fdf54bb9-f735-4810-82ee-d803099a1bcf-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fdf54bb9-f735-4810-82ee-d803099a1bcf-console-oauth-config\") pod \"fdf54bb9-f735-4810-82ee-d803099a1bcf\" (UID: \"fdf54bb9-f735-4810-82ee-d803099a1bcf\") "
Oct 30 21:51:14.216009 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:51:14.215930    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdf54bb9-f735-4810-82ee-d803099a1bcf-service-ca" pod "fdf54bb9-f735-4810-82ee-d803099a1bcf" (UID: "fdf54bb9-f735-4810-82ee-d803099a1bcf"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:51:14.216123 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:51:14.216090    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdf54bb9-f735-4810-82ee-d803099a1bcf-trusted-ca-bundle" pod "fdf54bb9-f735-4810-82ee-d803099a1bcf" (UID: "fdf54bb9-f735-4810-82ee-d803099a1bcf"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:51:14.216232 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:51:14.216121    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdf54bb9-f735-4810-82ee-d803099a1bcf-tmp" pod "fdf54bb9-f735-4810-82ee-d803099a1bcf" (UID: "fdf54bb9-f735-4810-82ee-d803099a1bcf"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:51:14.216232 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:51:14.216139    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdf54bb9-f735-4810-82ee-d803099a1bcf-oauth-serving-cert" pod "fdf54bb9-f735-4810-82ee-d803099a1bcf" (UID: "fdf54bb9-f735-4810-82ee-d803099a1bcf"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:51:14.216232 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:51:14.216149    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdf54bb9-f735-4810-82ee-d803099a1bcf-console-config" pod "fdf54bb9-f735-4810-82ee-d803099a1bcf" (UID: "fdf54bb9-f735-4810-82ee-d803099a1bcf"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:51:14.217727 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:51:14.217695    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdf54bb9-f735-4810-82ee-d803099a1bcf-console-serving-cert" pod "fdf54bb9-f735-4810-82ee-d803099a1bcf" (UID: "fdf54bb9-f735-4810-82ee-d803099a1bcf"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:51:14.217878 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:51:14.217856    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdf54bb9-f735-4810-82ee-d803099a1bcf-kube-api-access-s5jdv" pod "fdf54bb9-f735-4810-82ee-d803099a1bcf" (UID: "fdf54bb9-f735-4810-82ee-d803099a1bcf"). InnerVolumeSpecName "kube-api-access-s5jdv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:51:14.217980 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:51:14.217962    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdf54bb9-f735-4810-82ee-d803099a1bcf-console-oauth-config" pod "fdf54bb9-f735-4810-82ee-d803099a1bcf" (UID: "fdf54bb9-f735-4810-82ee-d803099a1bcf"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:51:14.316195 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:51:14.316164    5678 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fdf54bb9-f735-4810-82ee-d803099a1bcf-tmp\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:51:14.316195 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:51:14.316188    5678 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fdf54bb9-f735-4810-82ee-d803099a1bcf-console-oauth-config\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:51:14.316195 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:51:14.316199    5678 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fdf54bb9-f735-4810-82ee-d803099a1bcf-service-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:51:14.316394 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:51:14.316209    5678 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fdf54bb9-f735-4810-82ee-d803099a1bcf-console-serving-cert\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:51:14.316394 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:51:14.316219    5678 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fdf54bb9-f735-4810-82ee-d803099a1bcf-trusted-ca-bundle\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:51:14.316394 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:51:14.316227    5678 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fdf54bb9-f735-4810-82ee-d803099a1bcf-console-config\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:51:14.316394 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:51:14.316236    5678 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fdf54bb9-f735-4810-82ee-d803099a1bcf-oauth-serving-cert\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:51:14.316394 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:51:14.316243    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5jdv\" (UniqueName: \"kubernetes.io/projected/fdf54bb9-f735-4810-82ee-d803099a1bcf-kube-api-access-s5jdv\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:51:14.788007 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:51:14.787922    5678 generic.go:356] "Generic (PLEG): container finished" podID="fdf54bb9-f735-4810-82ee-d803099a1bcf" containerID="e39713154e1dc737f4cfcb4c40f6f2c41e5c329785e99738a71e93998afec246" exitCode=0
Oct 30 21:51:14.788007 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:51:14.787967    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7cbdbcf4b5-rmbsz"
Oct 30 21:51:14.788007 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:51:14.787979    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7cbdbcf4b5-rmbsz" event={"ID":"fdf54bb9-f735-4810-82ee-d803099a1bcf","Type":"ContainerDied","Data":"e39713154e1dc737f4cfcb4c40f6f2c41e5c329785e99738a71e93998afec246"}
Oct 30 21:51:14.788316 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:51:14.788019    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7cbdbcf4b5-rmbsz" event={"ID":"fdf54bb9-f735-4810-82ee-d803099a1bcf","Type":"ContainerDied","Data":"b958496c5215e685901b3b9da29749f1fd2fec924a1885b5175591646903879d"}
Oct 30 21:51:14.788316 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:51:14.788048    5678 scope.go:122] "RemoveContainer" containerID="e39713154e1dc737f4cfcb4c40f6f2c41e5c329785e99738a71e93998afec246"
Oct 30 21:51:14.795586 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:51:14.795567    5678 scope.go:122] "RemoveContainer" containerID="e39713154e1dc737f4cfcb4c40f6f2c41e5c329785e99738a71e93998afec246"
Oct 30 21:51:14.795841 ip-10-0-128-245 kubenswrapper[5678]: E1030 21:51:14.795823    5678 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e39713154e1dc737f4cfcb4c40f6f2c41e5c329785e99738a71e93998afec246\": container with ID starting with e39713154e1dc737f4cfcb4c40f6f2c41e5c329785e99738a71e93998afec246 not found: ID does not exist" containerID="e39713154e1dc737f4cfcb4c40f6f2c41e5c329785e99738a71e93998afec246"
Oct 30 21:51:14.795882 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:51:14.795849    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e39713154e1dc737f4cfcb4c40f6f2c41e5c329785e99738a71e93998afec246"} err="failed to get container status \"e39713154e1dc737f4cfcb4c40f6f2c41e5c329785e99738a71e93998afec246\": rpc error: code = NotFound desc = could not find container \"e39713154e1dc737f4cfcb4c40f6f2c41e5c329785e99738a71e93998afec246\": container with ID starting with e39713154e1dc737f4cfcb4c40f6f2c41e5c329785e99738a71e93998afec246 not found: ID does not exist"
Oct 30 21:51:14.943419 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:51:14.943380    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7cbdbcf4b5-rmbsz"]
Oct 30 21:51:14.990361 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:51:14.990334    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7cbdbcf4b5-rmbsz"]
Oct 30 21:51:16.520503 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:51:16.520469    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdf54bb9-f735-4810-82ee-d803099a1bcf" path="/var/lib/kubelet/pods/fdf54bb9-f735-4810-82ee-d803099a1bcf/volumes"
Oct 30 21:51:23.792416 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:51:23.792368    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:51:23.807525 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:51:23.807500    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:51:23.833833 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:51:23.833807    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:51:40.569199 ip-10-0-128-245 kubenswrapper[5678]: E1030 21:51:40.569163    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 21:52:10.569108 ip-10-0-128-245 kubenswrapper[5678]: E1030 21:52:10.569031    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 21:52:40.570827 ip-10-0-128-245 kubenswrapper[5678]: E1030 21:52:40.570792    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 21:53:10.570277 ip-10-0-128-245 kubenswrapper[5678]: E1030 21:53:10.570234    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 21:53:40.570126 ip-10-0-128-245 kubenswrapper[5678]: E1030 21:53:40.570047    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 21:54:10.570135 ip-10-0-128-245 kubenswrapper[5678]: E1030 21:54:10.570100    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 21:54:40.570133 ip-10-0-128-245 kubenswrapper[5678]: E1030 21:54:40.570093    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 21:55:10.570005 ip-10-0-128-245 kubenswrapper[5678]: E1030 21:55:10.569926    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 21:55:40.570294 ip-10-0-128-245 kubenswrapper[5678]: E1030 21:55:40.570261    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 21:56:10.569922 ip-10-0-128-245 kubenswrapper[5678]: E1030 21:56:10.569878    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 21:56:27.029387 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:27.029354    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csvbt2"]
Oct 30 21:56:27.029949 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:27.029653    5678 cpu_manager.go:414] "containerMap: removing container" podUID="fdf54bb9-f735-4810-82ee-d803099a1bcf" containerName="console"
Oct 30 21:56:27.029949 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:27.029664    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdf54bb9-f735-4810-82ee-d803099a1bcf" containerName="console"
Oct 30 21:56:27.029949 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:27.029675    5678 cpu_manager.go:414] "containerMap: removing container" podUID="69e218ec-ab5e-4e09-8444-e53a710c53bf" containerName="console"
Oct 30 21:56:27.029949 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:27.029680    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69e218ec-ab5e-4e09-8444-e53a710c53bf" containerName="console"
Oct 30 21:56:27.029949 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:27.029730    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdf54bb9-f735-4810-82ee-d803099a1bcf" containerName="console"
Oct 30 21:56:27.029949 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:27.029739    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="69e218ec-ab5e-4e09-8444-e53a710c53bf" containerName="console"
Oct 30 21:56:27.031896 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:27.031881    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csvbt2"
Oct 30 21:56:27.128866 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:27.128835    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Oct 30 21:56:27.128866 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:27.128859    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Oct 30 21:56:27.129077 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:27.128949    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-8pg2j\""
Oct 30 21:56:27.129464 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:27.129447    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csvbt2"]
Oct 30 21:56:27.168185 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:27.168146    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a2847b05-a3f6-430c-9052-3b73934599f1-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csvbt2\" (UID: \"a2847b05-a3f6-430c-9052-3b73934599f1\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csvbt2"
Oct 30 21:56:27.168332 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:27.168202    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a2847b05-a3f6-430c-9052-3b73934599f1-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csvbt2\" (UID: \"a2847b05-a3f6-430c-9052-3b73934599f1\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csvbt2"
Oct 30 21:56:27.168332 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:27.168290    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5gdj\" (UniqueName: \"kubernetes.io/projected/a2847b05-a3f6-430c-9052-3b73934599f1-kube-api-access-v5gdj\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csvbt2\" (UID: \"a2847b05-a3f6-430c-9052-3b73934599f1\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csvbt2"
Oct 30 21:56:27.269252 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:27.269224    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a2847b05-a3f6-430c-9052-3b73934599f1-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csvbt2\" (UID: \"a2847b05-a3f6-430c-9052-3b73934599f1\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csvbt2"
Oct 30 21:56:27.269444 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:27.269273    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5gdj\" (UniqueName: \"kubernetes.io/projected/a2847b05-a3f6-430c-9052-3b73934599f1-kube-api-access-v5gdj\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csvbt2\" (UID: \"a2847b05-a3f6-430c-9052-3b73934599f1\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csvbt2"
Oct 30 21:56:27.269444 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:27.269317    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a2847b05-a3f6-430c-9052-3b73934599f1-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csvbt2\" (UID: \"a2847b05-a3f6-430c-9052-3b73934599f1\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csvbt2"
Oct 30 21:56:27.269636 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:27.269616    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a2847b05-a3f6-430c-9052-3b73934599f1-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csvbt2\" (UID: \"a2847b05-a3f6-430c-9052-3b73934599f1\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csvbt2"
Oct 30 21:56:27.269672 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:27.269647    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a2847b05-a3f6-430c-9052-3b73934599f1-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csvbt2\" (UID: \"a2847b05-a3f6-430c-9052-3b73934599f1\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csvbt2"
Oct 30 21:56:27.328765 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:27.328705    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5gdj\" (UniqueName: \"kubernetes.io/projected/a2847b05-a3f6-430c-9052-3b73934599f1-kube-api-access-v5gdj\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csvbt2\" (UID: \"a2847b05-a3f6-430c-9052-3b73934599f1\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csvbt2"
Oct 30 21:56:27.344368 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:27.344346    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csvbt2"
Oct 30 21:56:27.716978 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:27.716951    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csvbt2"]
Oct 30 21:56:27.720081 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:56:27.720052    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2847b05_a3f6_430c_9052_3b73934599f1.slice/crio-0a68be6575357ede071e99a5555866c96dfe00ec3f45e604df04b58e714850a9 WatchSource:0}: Error finding container 0a68be6575357ede071e99a5555866c96dfe00ec3f45e604df04b58e714850a9: Status 404 returned error can't find the container with id 0a68be6575357ede071e99a5555866c96dfe00ec3f45e604df04b58e714850a9
Oct 30 21:56:27.722132 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:27.722115    5678 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 21:56:27.773696 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:27.773663    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csvbt2" event={"ID":"a2847b05-a3f6-430c-9052-3b73934599f1","Type":"ContainerStarted","Data":"0a68be6575357ede071e99a5555866c96dfe00ec3f45e604df04b58e714850a9"}
Oct 30 21:56:28.622046 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:28.622016    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mzkjf"]
Oct 30 21:56:28.625018 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:28.625001    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mzkjf"
Oct 30 21:56:28.681082 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:28.681034    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d1cde967-3ef6-4c48-bef1-59574f7f2479-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mzkjf\" (UID: \"d1cde967-3ef6-4c48-bef1-59574f7f2479\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mzkjf"
Oct 30 21:56:28.681261 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:28.681104    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7p2qw\" (UniqueName: \"kubernetes.io/projected/d1cde967-3ef6-4c48-bef1-59574f7f2479-kube-api-access-7p2qw\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mzkjf\" (UID: \"d1cde967-3ef6-4c48-bef1-59574f7f2479\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mzkjf"
Oct 30 21:56:28.681344 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:28.681238    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d1cde967-3ef6-4c48-bef1-59574f7f2479-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mzkjf\" (UID: \"d1cde967-3ef6-4c48-bef1-59574f7f2479\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mzkjf"
Oct 30 21:56:28.721114 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:28.721091    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-9f7w5\""
Oct 30 21:56:28.721718 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:28.721702    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mzkjf"]
Oct 30 21:56:28.722022 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:28.722005    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d788crs"]
Oct 30 21:56:28.724419 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:28.724392    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d788crs"
Oct 30 21:56:28.776199 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:28.776167    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-j4xfx\""
Oct 30 21:56:28.780832 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:28.780801    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d788crs"]
Oct 30 21:56:28.782462 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:28.782439    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d1cde967-3ef6-4c48-bef1-59574f7f2479-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mzkjf\" (UID: \"d1cde967-3ef6-4c48-bef1-59574f7f2479\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mzkjf"
Oct 30 21:56:28.782587 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:28.782482    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7p2qw\" (UniqueName: \"kubernetes.io/projected/d1cde967-3ef6-4c48-bef1-59574f7f2479-kube-api-access-7p2qw\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mzkjf\" (UID: \"d1cde967-3ef6-4c48-bef1-59574f7f2479\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mzkjf"
Oct 30 21:56:28.782686 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:28.782670    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v549x\" (UniqueName: \"kubernetes.io/projected/8eae4463-6286-4817-9319-bc4b0d528ede-kube-api-access-v549x\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d788crs\" (UID: \"8eae4463-6286-4817-9319-bc4b0d528ede\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d788crs"
Oct 30 21:56:28.782749 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:28.782698    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d1cde967-3ef6-4c48-bef1-59574f7f2479-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mzkjf\" (UID: \"d1cde967-3ef6-4c48-bef1-59574f7f2479\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mzkjf"
Oct 30 21:56:28.782749 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:28.782721    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8eae4463-6286-4817-9319-bc4b0d528ede-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d788crs\" (UID: \"8eae4463-6286-4817-9319-bc4b0d528ede\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d788crs"
Oct 30 21:56:28.782852 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:28.782803    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d1cde967-3ef6-4c48-bef1-59574f7f2479-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mzkjf\" (UID: \"d1cde967-3ef6-4c48-bef1-59574f7f2479\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mzkjf"
Oct 30 21:56:28.782909 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:28.782893    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8eae4463-6286-4817-9319-bc4b0d528ede-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d788crs\" (UID: \"8eae4463-6286-4817-9319-bc4b0d528ede\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d788crs"
Oct 30 21:56:28.783061 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:28.783041    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d1cde967-3ef6-4c48-bef1-59574f7f2479-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mzkjf\" (UID: \"d1cde967-3ef6-4c48-bef1-59574f7f2479\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mzkjf"
Oct 30 21:56:28.827024 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:28.827001    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7lltl"]
Oct 30 21:56:28.829448 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:28.829428    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7lltl"
Oct 30 21:56:28.841237 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:28.841214    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7p2qw\" (UniqueName: \"kubernetes.io/projected/d1cde967-3ef6-4c48-bef1-59574f7f2479-kube-api-access-7p2qw\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mzkjf\" (UID: \"d1cde967-3ef6-4c48-bef1-59574f7f2479\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mzkjf"
Oct 30 21:56:28.879547 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:28.879464    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-569zq\""
Oct 30 21:56:28.883641 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:28.883620    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8eae4463-6286-4817-9319-bc4b0d528ede-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d788crs\" (UID: \"8eae4463-6286-4817-9319-bc4b0d528ede\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d788crs"
Oct 30 21:56:28.883738 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:28.883660    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ee340d2f-4dc4-4d12-9f68-62402f7916b8-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7lltl\" (UID: \"ee340d2f-4dc4-4d12-9f68-62402f7916b8\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7lltl"
Oct 30 21:56:28.883738 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:28.883686    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8eae4463-6286-4817-9319-bc4b0d528ede-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d788crs\" (UID: \"8eae4463-6286-4817-9319-bc4b0d528ede\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d788crs"
Oct 30 21:56:28.883828 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:28.883762    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfbvf\" (UniqueName: \"kubernetes.io/projected/ee340d2f-4dc4-4d12-9f68-62402f7916b8-kube-api-access-qfbvf\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7lltl\" (UID: \"ee340d2f-4dc4-4d12-9f68-62402f7916b8\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7lltl"
Oct 30 21:56:28.883828 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:28.883820    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ee340d2f-4dc4-4d12-9f68-62402f7916b8-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7lltl\" (UID: \"ee340d2f-4dc4-4d12-9f68-62402f7916b8\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7lltl"
Oct 30 21:56:28.883927 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:28.883868    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v549x\" (UniqueName: \"kubernetes.io/projected/8eae4463-6286-4817-9319-bc4b0d528ede-kube-api-access-v549x\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d788crs\" (UID: \"8eae4463-6286-4817-9319-bc4b0d528ede\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d788crs"
Oct 30 21:56:28.884047 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:28.884028    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8eae4463-6286-4817-9319-bc4b0d528ede-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d788crs\" (UID: \"8eae4463-6286-4817-9319-bc4b0d528ede\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d788crs"
Oct 30 21:56:28.884088 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:28.884065    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8eae4463-6286-4817-9319-bc4b0d528ede-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d788crs\" (UID: \"8eae4463-6286-4817-9319-bc4b0d528ede\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d788crs"
Oct 30 21:56:28.886815 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:28.886792    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7lltl"]
Oct 30 21:56:28.919154 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:28.919124    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpdjqd"]
Oct 30 21:56:28.921582 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:28.921560    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpdjqd"
Oct 30 21:56:28.938821 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:28.938802    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mzkjf"
Oct 30 21:56:28.942508 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:28.942486    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v549x\" (UniqueName: \"kubernetes.io/projected/8eae4463-6286-4817-9319-bc4b0d528ede-kube-api-access-v549x\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d788crs\" (UID: \"8eae4463-6286-4817-9319-bc4b0d528ede\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d788crs"
Oct 30 21:56:28.978790 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:28.978722    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpdjqd"]
Oct 30 21:56:28.980034 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:28.980010    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-5rx8w\""
Oct 30 21:56:28.985324 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:28.985293    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfbvf\" (UniqueName: \"kubernetes.io/projected/ee340d2f-4dc4-4d12-9f68-62402f7916b8-kube-api-access-qfbvf\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7lltl\" (UID: \"ee340d2f-4dc4-4d12-9f68-62402f7916b8\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7lltl"
Oct 30 21:56:28.985433 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:28.985350    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ee340d2f-4dc4-4d12-9f68-62402f7916b8-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7lltl\" (UID: \"ee340d2f-4dc4-4d12-9f68-62402f7916b8\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7lltl"
Oct 30 21:56:28.985495 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:28.985433    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/76bac5b5-d258-4942-8a65-b0ef3f55b55b-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpdjqd\" (UID: \"76bac5b5-d258-4942-8a65-b0ef3f55b55b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpdjqd"
Oct 30 21:56:28.985495 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:28.985475    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/76bac5b5-d258-4942-8a65-b0ef3f55b55b-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpdjqd\" (UID: \"76bac5b5-d258-4942-8a65-b0ef3f55b55b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpdjqd"
Oct 30 21:56:28.985591 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:28.985508    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ee340d2f-4dc4-4d12-9f68-62402f7916b8-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7lltl\" (UID: \"ee340d2f-4dc4-4d12-9f68-62402f7916b8\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7lltl"
Oct 30 21:56:28.985591 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:28.985549    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jxgs\" (UniqueName: \"kubernetes.io/projected/76bac5b5-d258-4942-8a65-b0ef3f55b55b-kube-api-access-9jxgs\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpdjqd\" (UID: \"76bac5b5-d258-4942-8a65-b0ef3f55b55b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpdjqd"
Oct 30 21:56:28.986317 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:28.986269    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ee340d2f-4dc4-4d12-9f68-62402f7916b8-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7lltl\" (UID: \"ee340d2f-4dc4-4d12-9f68-62402f7916b8\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7lltl"
Oct 30 21:56:28.986524 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:28.986501    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ee340d2f-4dc4-4d12-9f68-62402f7916b8-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7lltl\" (UID: \"ee340d2f-4dc4-4d12-9f68-62402f7916b8\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7lltl"
Oct 30 21:56:29.037489 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:29.037460    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d788crs"
Oct 30 21:56:29.044248 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:29.044215    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfbvf\" (UniqueName: \"kubernetes.io/projected/ee340d2f-4dc4-4d12-9f68-62402f7916b8-kube-api-access-qfbvf\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7lltl\" (UID: \"ee340d2f-4dc4-4d12-9f68-62402f7916b8\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7lltl"
Oct 30 21:56:29.086109 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:29.086076    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/76bac5b5-d258-4942-8a65-b0ef3f55b55b-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpdjqd\" (UID: \"76bac5b5-d258-4942-8a65-b0ef3f55b55b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpdjqd"
Oct 30 21:56:29.086282 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:29.086122    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/76bac5b5-d258-4942-8a65-b0ef3f55b55b-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpdjqd\" (UID: \"76bac5b5-d258-4942-8a65-b0ef3f55b55b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpdjqd"
Oct 30 21:56:29.086282 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:29.086160    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jxgs\" (UniqueName: \"kubernetes.io/projected/76bac5b5-d258-4942-8a65-b0ef3f55b55b-kube-api-access-9jxgs\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpdjqd\" (UID: \"76bac5b5-d258-4942-8a65-b0ef3f55b55b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpdjqd"
Oct 30 21:56:29.086470 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:29.086452    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/76bac5b5-d258-4942-8a65-b0ef3f55b55b-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpdjqd\" (UID: \"76bac5b5-d258-4942-8a65-b0ef3f55b55b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpdjqd"
Oct 30 21:56:29.086546 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:29.086510    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/76bac5b5-d258-4942-8a65-b0ef3f55b55b-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpdjqd\" (UID: \"76bac5b5-d258-4942-8a65-b0ef3f55b55b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpdjqd"
Oct 30 21:56:29.143243 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:29.143148    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7lltl"
Oct 30 21:56:29.145635 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:29.145608    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jxgs\" (UniqueName: \"kubernetes.io/projected/76bac5b5-d258-4942-8a65-b0ef3f55b55b-kube-api-access-9jxgs\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpdjqd\" (UID: \"76bac5b5-d258-4942-8a65-b0ef3f55b55b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpdjqd"
Oct 30 21:56:29.235684 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:29.235657    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpdjqd"
Oct 30 21:56:29.313976 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:29.313951    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mzkjf"]
Oct 30 21:56:29.316890 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:56:29.316862    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1cde967_3ef6_4c48_bef1_59574f7f2479.slice/crio-af2dce08c33df29c5ac63fee7b833b87083b26a47c93f718312e54db74914e9d WatchSource:0}: Error finding container af2dce08c33df29c5ac63fee7b833b87083b26a47c93f718312e54db74914e9d: Status 404 returned error can't find the container with id af2dce08c33df29c5ac63fee7b833b87083b26a47c93f718312e54db74914e9d
Oct 30 21:56:29.423177 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:29.423096    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d788crs"]
Oct 30 21:56:29.427267 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:56:29.427240    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8eae4463_6286_4817_9319_bc4b0d528ede.slice/crio-c548f77a1149bdcffba8903671d5f68385b565a47f65f3b868c66479c68f62da WatchSource:0}: Error finding container c548f77a1149bdcffba8903671d5f68385b565a47f65f3b868c66479c68f62da: Status 404 returned error can't find the container with id c548f77a1149bdcffba8903671d5f68385b565a47f65f3b868c66479c68f62da
Oct 30 21:56:29.545086 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:29.545050    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7lltl"]
Oct 30 21:56:29.548590 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:56:29.548554    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee340d2f_4dc4_4d12_9f68_62402f7916b8.slice/crio-a7be542e6ecb24fedb99473a332dde495aa29db76976947f3785b6707ce94d88 WatchSource:0}: Error finding container a7be542e6ecb24fedb99473a332dde495aa29db76976947f3785b6707ce94d88: Status 404 returned error can't find the container with id a7be542e6ecb24fedb99473a332dde495aa29db76976947f3785b6707ce94d88
Oct 30 21:56:29.611278 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:29.611252    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpdjqd"]
Oct 30 21:56:29.615896 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:56:29.615867    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76bac5b5_d258_4942_8a65_b0ef3f55b55b.slice/crio-7724e6d8272b8233a9da0bfca53d4127a4b624eba23e388d8870c8b19ec34ccd WatchSource:0}: Error finding container 7724e6d8272b8233a9da0bfca53d4127a4b624eba23e388d8870c8b19ec34ccd: Status 404 returned error can't find the container with id 7724e6d8272b8233a9da0bfca53d4127a4b624eba23e388d8870c8b19ec34ccd
Oct 30 21:56:29.781213 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:29.781130    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7lltl" event={"ID":"ee340d2f-4dc4-4d12-9f68-62402f7916b8","Type":"ContainerStarted","Data":"a7be542e6ecb24fedb99473a332dde495aa29db76976947f3785b6707ce94d88"}
Oct 30 21:56:29.782465 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:29.782437    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d788crs" event={"ID":"8eae4463-6286-4817-9319-bc4b0d528ede","Type":"ContainerStarted","Data":"c548f77a1149bdcffba8903671d5f68385b565a47f65f3b868c66479c68f62da"}
Oct 30 21:56:29.783584 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:29.783560    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mzkjf" event={"ID":"d1cde967-3ef6-4c48-bef1-59574f7f2479","Type":"ContainerStarted","Data":"af2dce08c33df29c5ac63fee7b833b87083b26a47c93f718312e54db74914e9d"}
Oct 30 21:56:29.784590 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:29.784567    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpdjqd" event={"ID":"76bac5b5-d258-4942-8a65-b0ef3f55b55b","Type":"ContainerStarted","Data":"7724e6d8272b8233a9da0bfca53d4127a4b624eba23e388d8870c8b19ec34ccd"}
Oct 30 21:56:30.062473 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:30.062367    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644nj4bh"]
Oct 30 21:56:30.066246 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:30.066222    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644nj4bh"
Oct 30 21:56:30.115869 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:30.115841    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-fmpt6\""
Oct 30 21:56:30.122360 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:30.122331    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644nj4bh"]
Oct 30 21:56:30.195999 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:30.195968    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmhnr\" (UniqueName: \"kubernetes.io/projected/105ffac7-32e6-415c-bb60-235b2d822169-kube-api-access-pmhnr\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644nj4bh\" (UID: \"105ffac7-32e6-415c-bb60-235b2d822169\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644nj4bh"
Oct 30 21:56:30.196189 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:30.196042    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/105ffac7-32e6-415c-bb60-235b2d822169-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644nj4bh\" (UID: \"105ffac7-32e6-415c-bb60-235b2d822169\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644nj4bh"
Oct 30 21:56:30.196189 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:30.196146    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/105ffac7-32e6-415c-bb60-235b2d822169-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644nj4bh\" (UID: \"105ffac7-32e6-415c-bb60-235b2d822169\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644nj4bh"
Oct 30 21:56:30.297433 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:30.297379    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/105ffac7-32e6-415c-bb60-235b2d822169-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644nj4bh\" (UID: \"105ffac7-32e6-415c-bb60-235b2d822169\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644nj4bh"
Oct 30 21:56:30.297577 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:30.297453    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmhnr\" (UniqueName: \"kubernetes.io/projected/105ffac7-32e6-415c-bb60-235b2d822169-kube-api-access-pmhnr\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644nj4bh\" (UID: \"105ffac7-32e6-415c-bb60-235b2d822169\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644nj4bh"
Oct 30 21:56:30.297577 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:30.297529    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/105ffac7-32e6-415c-bb60-235b2d822169-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644nj4bh\" (UID: \"105ffac7-32e6-415c-bb60-235b2d822169\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644nj4bh"
Oct 30 21:56:30.297826 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:30.297722    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/105ffac7-32e6-415c-bb60-235b2d822169-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644nj4bh\" (UID: \"105ffac7-32e6-415c-bb60-235b2d822169\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644nj4bh"
Oct 30 21:56:30.297826 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:30.297789    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/105ffac7-32e6-415c-bb60-235b2d822169-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644nj4bh\" (UID: \"105ffac7-32e6-415c-bb60-235b2d822169\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644nj4bh"
Oct 30 21:56:30.356840 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:30.356741    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmhnr\" (UniqueName: \"kubernetes.io/projected/105ffac7-32e6-415c-bb60-235b2d822169-kube-api-access-pmhnr\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644nj4bh\" (UID: \"105ffac7-32e6-415c-bb60-235b2d822169\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644nj4bh"
Oct 30 21:56:30.382536 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:30.382509    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644nj4bh"
Oct 30 21:56:30.766344 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:30.766314    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644nj4bh"]
Oct 30 21:56:30.772912 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:56:30.772875    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod105ffac7_32e6_415c_bb60_235b2d822169.slice/crio-f4234612ff9d8dd5026435b1a24e678866fc8650a4a2ec297e1570db81b3685d WatchSource:0}: Error finding container f4234612ff9d8dd5026435b1a24e678866fc8650a4a2ec297e1570db81b3685d: Status 404 returned error can't find the container with id f4234612ff9d8dd5026435b1a24e678866fc8650a4a2ec297e1570db81b3685d
Oct 30 21:56:30.790746 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:30.790661    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644nj4bh" event={"ID":"105ffac7-32e6-415c-bb60-235b2d822169","Type":"ContainerStarted","Data":"f4234612ff9d8dd5026435b1a24e678866fc8650a4a2ec297e1570db81b3685d"}
Oct 30 21:56:31.055510 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:31.055432    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714527g"]
Oct 30 21:56:31.059279 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:31.058538    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714527g"
Oct 30 21:56:31.112013 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:31.111980    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-zbtc4\""
Oct 30 21:56:31.115599 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:31.115573    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714527g"]
Oct 30 21:56:31.207233 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:31.206908    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rpzb\" (UniqueName: \"kubernetes.io/projected/4bfc7de7-025b-4746-847c-7be0eba28b14-kube-api-access-6rpzb\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714527g\" (UID: \"4bfc7de7-025b-4746-847c-7be0eba28b14\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714527g"
Oct 30 21:56:31.207233 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:31.206978    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4bfc7de7-025b-4746-847c-7be0eba28b14-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714527g\" (UID: \"4bfc7de7-025b-4746-847c-7be0eba28b14\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714527g"
Oct 30 21:56:31.207233 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:31.207041    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4bfc7de7-025b-4746-847c-7be0eba28b14-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714527g\" (UID: \"4bfc7de7-025b-4746-847c-7be0eba28b14\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714527g"
Oct 30 21:56:31.307598 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:31.307512    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4bfc7de7-025b-4746-847c-7be0eba28b14-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714527g\" (UID: \"4bfc7de7-025b-4746-847c-7be0eba28b14\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714527g"
Oct 30 21:56:31.307598 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:31.307582    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4bfc7de7-025b-4746-847c-7be0eba28b14-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714527g\" (UID: \"4bfc7de7-025b-4746-847c-7be0eba28b14\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714527g"
Oct 30 21:56:31.307834 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:31.307642    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rpzb\" (UniqueName: \"kubernetes.io/projected/4bfc7de7-025b-4746-847c-7be0eba28b14-kube-api-access-6rpzb\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714527g\" (UID: \"4bfc7de7-025b-4746-847c-7be0eba28b14\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714527g"
Oct 30 21:56:31.308369 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:31.308341    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4bfc7de7-025b-4746-847c-7be0eba28b14-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714527g\" (UID: \"4bfc7de7-025b-4746-847c-7be0eba28b14\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714527g"
Oct 30 21:56:31.308628 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:31.308607    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4bfc7de7-025b-4746-847c-7be0eba28b14-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714527g\" (UID: \"4bfc7de7-025b-4746-847c-7be0eba28b14\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714527g"
Oct 30 21:56:31.370247 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:31.370196    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rpzb\" (UniqueName: \"kubernetes.io/projected/4bfc7de7-025b-4746-847c-7be0eba28b14-kube-api-access-6rpzb\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714527g\" (UID: \"4bfc7de7-025b-4746-847c-7be0eba28b14\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714527g"
Oct 30 21:56:31.378002 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:31.377941    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714527g"
Oct 30 21:56:31.772722 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:31.772640    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714527g"]
Oct 30 21:56:31.776634 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:56:31.776595    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bfc7de7_025b_4746_847c_7be0eba28b14.slice/crio-7049dcf0b720d8a80d3b98febac7f5224a3e15784ab6df099d2f3894b57257c2 WatchSource:0}: Error finding container 7049dcf0b720d8a80d3b98febac7f5224a3e15784ab6df099d2f3894b57257c2: Status 404 returned error can't find the container with id 7049dcf0b720d8a80d3b98febac7f5224a3e15784ab6df099d2f3894b57257c2
Oct 30 21:56:31.798730 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:31.798697    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714527g" event={"ID":"4bfc7de7-025b-4746-847c-7be0eba28b14","Type":"ContainerStarted","Data":"7049dcf0b720d8a80d3b98febac7f5224a3e15784ab6df099d2f3894b57257c2"}
Oct 30 21:56:40.608451 ip-10-0-128-245 kubenswrapper[5678]: E1030 21:56:40.608363    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 21:56:41.887430 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:41.886825    5678 generic.go:356] "Generic (PLEG): container finished" podID="a2847b05-a3f6-430c-9052-3b73934599f1" containerID="e8d9c01da9b604de680cfc1f63e7c8a82c92e62e3e5750972a57c3f27cd370dd" exitCode=0
Oct 30 21:56:41.887430 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:41.886948    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csvbt2" event={"ID":"a2847b05-a3f6-430c-9052-3b73934599f1","Type":"ContainerDied","Data":"e8d9c01da9b604de680cfc1f63e7c8a82c92e62e3e5750972a57c3f27cd370dd"}
Oct 30 21:56:43.898555 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:43.898519    5678 generic.go:356] "Generic (PLEG): container finished" podID="8eae4463-6286-4817-9319-bc4b0d528ede" containerID="0e41688676c0128bee846758d0a2f165b7d0f3f207f3a7662f2613388240f48b" exitCode=0
Oct 30 21:56:43.898968 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:43.898607    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d788crs" event={"ID":"8eae4463-6286-4817-9319-bc4b0d528ede","Type":"ContainerDied","Data":"0e41688676c0128bee846758d0a2f165b7d0f3f207f3a7662f2613388240f48b"}
Oct 30 21:56:43.901323 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:43.901299    5678 generic.go:356] "Generic (PLEG): container finished" podID="d1cde967-3ef6-4c48-bef1-59574f7f2479" containerID="dd538a6c1ed1cdeb3da6d5fabe8e3c044b7b4e77c7793059a43ba6c643a26ec6" exitCode=0
Oct 30 21:56:43.901649 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:43.901425    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mzkjf" event={"ID":"d1cde967-3ef6-4c48-bef1-59574f7f2479","Type":"ContainerDied","Data":"dd538a6c1ed1cdeb3da6d5fabe8e3c044b7b4e77c7793059a43ba6c643a26ec6"}
Oct 30 21:56:43.903518 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:43.903492    5678 generic.go:356] "Generic (PLEG): container finished" podID="76bac5b5-d258-4942-8a65-b0ef3f55b55b" containerID="296cd591441ef5161b38ce66787c75dedd4044f5e45d41f99525794d8e899023" exitCode=0
Oct 30 21:56:43.903623 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:43.903569    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpdjqd" event={"ID":"76bac5b5-d258-4942-8a65-b0ef3f55b55b","Type":"ContainerDied","Data":"296cd591441ef5161b38ce66787c75dedd4044f5e45d41f99525794d8e899023"}
Oct 30 21:56:43.906648 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:43.906623    5678 generic.go:356] "Generic (PLEG): container finished" podID="ee340d2f-4dc4-4d12-9f68-62402f7916b8" containerID="301ee34a9fcb676d577e31c6e6bf36582f19cbead22e5bd824efec1fc8b077c0" exitCode=0
Oct 30 21:56:43.906752 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:43.906666    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7lltl" event={"ID":"ee340d2f-4dc4-4d12-9f68-62402f7916b8","Type":"ContainerDied","Data":"301ee34a9fcb676d577e31c6e6bf36582f19cbead22e5bd824efec1fc8b077c0"}
Oct 30 21:56:44.915743 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:44.915704    5678 generic.go:356] "Generic (PLEG): container finished" podID="105ffac7-32e6-415c-bb60-235b2d822169" containerID="9b6bd19bfb93d975effaa6cffd54d4845bcb5992248eab2d94119adba77f9d98" exitCode=0
Oct 30 21:56:44.916215 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:44.915895    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644nj4bh" event={"ID":"105ffac7-32e6-415c-bb60-235b2d822169","Type":"ContainerDied","Data":"9b6bd19bfb93d975effaa6cffd54d4845bcb5992248eab2d94119adba77f9d98"}
Oct 30 21:56:44.918613 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:44.918572    5678 generic.go:356] "Generic (PLEG): container finished" podID="4bfc7de7-025b-4746-847c-7be0eba28b14" containerID="395e5e1d7b971139bc21ef2176bc004dd8bab7910b55ea298958e0faa83c3662" exitCode=0
Oct 30 21:56:44.918733 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:44.918715    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714527g" event={"ID":"4bfc7de7-025b-4746-847c-7be0eba28b14","Type":"ContainerDied","Data":"395e5e1d7b971139bc21ef2176bc004dd8bab7910b55ea298958e0faa83c3662"}
Oct 30 21:56:45.926280 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:45.926248    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714527g" event={"ID":"4bfc7de7-025b-4746-847c-7be0eba28b14","Type":"ContainerStarted","Data":"df34fe918c54ba0548e2504095ba1dcb70242636fd3d0329d27ff85a959628df"}
Oct 30 21:56:45.928115 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:45.928088    5678 generic.go:356] "Generic (PLEG): container finished" podID="a2847b05-a3f6-430c-9052-3b73934599f1" containerID="17217c5198c63b6fe883206dac0dcb0f9791f6ed336fcd9cd4c8b3167a8513dd" exitCode=0
Oct 30 21:56:45.928238 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:45.928127    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csvbt2" event={"ID":"a2847b05-a3f6-430c-9052-3b73934599f1","Type":"ContainerDied","Data":"17217c5198c63b6fe883206dac0dcb0f9791f6ed336fcd9cd4c8b3167a8513dd"}
Oct 30 21:56:45.933094 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:45.933072    5678 generic.go:356] "Generic (PLEG): container finished" podID="8eae4463-6286-4817-9319-bc4b0d528ede" containerID="12809374ac6a1229dc8aaa53bd9b2ec17b56b002c28a487fc20c1887c21025bd" exitCode=0
Oct 30 21:56:45.933194 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:45.933138    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d788crs" event={"ID":"8eae4463-6286-4817-9319-bc4b0d528ede","Type":"ContainerDied","Data":"12809374ac6a1229dc8aaa53bd9b2ec17b56b002c28a487fc20c1887c21025bd"}
Oct 30 21:56:45.936874 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:45.936558    5678 generic.go:356] "Generic (PLEG): container finished" podID="d1cde967-3ef6-4c48-bef1-59574f7f2479" containerID="bbbacdb10fc2a445f466e269b4573d5ba51fbdb8daef678bed0e2609bfb8d39a" exitCode=0
Oct 30 21:56:45.936874 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:45.936670    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mzkjf" event={"ID":"d1cde967-3ef6-4c48-bef1-59574f7f2479","Type":"ContainerDied","Data":"bbbacdb10fc2a445f466e269b4573d5ba51fbdb8daef678bed0e2609bfb8d39a"}
Oct 30 21:56:45.938958 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:45.938937    5678 generic.go:356] "Generic (PLEG): container finished" podID="76bac5b5-d258-4942-8a65-b0ef3f55b55b" containerID="4839058d4e8dcaf16583235a54cb08e450c1d6c3bb5fcfb9903f620500ccaf05" exitCode=0
Oct 30 21:56:45.939043 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:45.938975    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpdjqd" event={"ID":"76bac5b5-d258-4942-8a65-b0ef3f55b55b","Type":"ContainerDied","Data":"4839058d4e8dcaf16583235a54cb08e450c1d6c3bb5fcfb9903f620500ccaf05"}
Oct 30 21:56:46.946330 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:46.946294    5678 generic.go:356] "Generic (PLEG): container finished" podID="ee340d2f-4dc4-4d12-9f68-62402f7916b8" containerID="0c9cc5886fe3aa98988255dc7ce9ba0fac832e930f866f8975cf63cbeb078b17" exitCode=0
Oct 30 21:56:46.946767 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:46.946377    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7lltl" event={"ID":"ee340d2f-4dc4-4d12-9f68-62402f7916b8","Type":"ContainerDied","Data":"0c9cc5886fe3aa98988255dc7ce9ba0fac832e930f866f8975cf63cbeb078b17"}
Oct 30 21:56:46.948819 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:46.948762    5678 generic.go:356] "Generic (PLEG): container finished" podID="4bfc7de7-025b-4746-847c-7be0eba28b14" containerID="df34fe918c54ba0548e2504095ba1dcb70242636fd3d0329d27ff85a959628df" exitCode=0
Oct 30 21:56:46.948819 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:46.948803    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714527g" event={"ID":"4bfc7de7-025b-4746-847c-7be0eba28b14","Type":"ContainerDied","Data":"df34fe918c54ba0548e2504095ba1dcb70242636fd3d0329d27ff85a959628df"}
Oct 30 21:56:49.975802 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:49.975764    5678 generic.go:356] "Generic (PLEG): container finished" podID="105ffac7-32e6-415c-bb60-235b2d822169" containerID="a96d6d359ea9c471f49b6c738f20efce5cbcdf3a748446e4872dcd72205fdb3d" exitCode=0
Oct 30 21:56:49.976360 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:56:49.975817    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644nj4bh" event={"ID":"105ffac7-32e6-415c-bb60-235b2d822169","Type":"ContainerDied","Data":"a96d6d359ea9c471f49b6c738f20efce5cbcdf3a748446e4872dcd72205fdb3d"}
Oct 30 21:57:02.047027 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:02.046995    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csvbt2" event={"ID":"a2847b05-a3f6-430c-9052-3b73934599f1","Type":"ContainerStarted","Data":"dab0cbc8d7e1f30d3e646ee27a5a26b7960d446d7747326f7b1a118434026cf9"}
Oct 30 21:57:02.049081 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:02.049053    5678 generic.go:356] "Generic (PLEG): container finished" podID="8eae4463-6286-4817-9319-bc4b0d528ede" containerID="30bbed195ba0f0a1aafbe6b97b2ac5e8ceef430e0583918ad05aaf6fe2802bcd" exitCode=0
Oct 30 21:57:02.049210 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:02.049141    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d788crs" event={"ID":"8eae4463-6286-4817-9319-bc4b0d528ede","Type":"ContainerDied","Data":"30bbed195ba0f0a1aafbe6b97b2ac5e8ceef430e0583918ad05aaf6fe2802bcd"}
Oct 30 21:57:02.051315 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:02.051297    5678 generic.go:356] "Generic (PLEG): container finished" podID="d1cde967-3ef6-4c48-bef1-59574f7f2479" containerID="d5c8975d4274d59f53b83c03c4c1690392ec8ea847ee81d647acf7829cdd6001" exitCode=0
Oct 30 21:57:02.051443 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:02.051423    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mzkjf" event={"ID":"d1cde967-3ef6-4c48-bef1-59574f7f2479","Type":"ContainerDied","Data":"d5c8975d4274d59f53b83c03c4c1690392ec8ea847ee81d647acf7829cdd6001"}
Oct 30 21:57:02.053490 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:02.053466    5678 generic.go:356] "Generic (PLEG): container finished" podID="76bac5b5-d258-4942-8a65-b0ef3f55b55b" containerID="547758e0e157bc5fd686123c1efc8fa0d8b2d25e94069db3df05675ad4ac1b7d" exitCode=0
Oct 30 21:57:02.053568 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:02.053533    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpdjqd" event={"ID":"76bac5b5-d258-4942-8a65-b0ef3f55b55b","Type":"ContainerDied","Data":"547758e0e157bc5fd686123c1efc8fa0d8b2d25e94069db3df05675ad4ac1b7d"}
Oct 30 21:57:02.055379 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:02.055361    5678 generic.go:356] "Generic (PLEG): container finished" podID="105ffac7-32e6-415c-bb60-235b2d822169" containerID="fe76636bb135c3e27341b61ba38bdc5e2124a0027a2c741ca955f70909a993fc" exitCode=0
Oct 30 21:57:02.055487 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:02.055461    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644nj4bh" event={"ID":"105ffac7-32e6-415c-bb60-235b2d822169","Type":"ContainerDied","Data":"fe76636bb135c3e27341b61ba38bdc5e2124a0027a2c741ca955f70909a993fc"}
Oct 30 21:57:02.057219 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:02.057200    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714527g" event={"ID":"4bfc7de7-025b-4746-847c-7be0eba28b14","Type":"ContainerStarted","Data":"91e0d8cc634f04d438f7feceb4da20bc97c051b3abc398d8e41f46c076be4027"}
Oct 30 21:57:02.191381 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:02.191333    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csvbt2" podStartSLOduration=3.431388657 podStartE2EDuration="36.191320602s" totalImagesPullingTime="32.759931945s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:56:26 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:57:02.188550726 +0000 UTC m=+399.792858279" watchObservedRunningTime="2026-10-30 21:57:02.191320602 +0000 UTC m=+399.795628149"
Oct 30 21:57:02.776771 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:02.776711    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714527g" podStartSLOduration=3.475166085 podStartE2EDuration="31.776692656s" totalImagesPullingTime="28.301526571s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:56:31 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:57:02.776549962 +0000 UTC m=+400.380857515" watchObservedRunningTime="2026-10-30 21:57:02.776692656 +0000 UTC m=+400.381000212"
Oct 30 21:57:03.063629 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:03.063550    5678 generic.go:356] "Generic (PLEG): container finished" podID="a2847b05-a3f6-430c-9052-3b73934599f1" containerID="dab0cbc8d7e1f30d3e646ee27a5a26b7960d446d7747326f7b1a118434026cf9" exitCode=0
Oct 30 21:57:03.064013 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:03.063633    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csvbt2" event={"ID":"a2847b05-a3f6-430c-9052-3b73934599f1","Type":"ContainerDied","Data":"dab0cbc8d7e1f30d3e646ee27a5a26b7960d446d7747326f7b1a118434026cf9"}
Oct 30 21:57:03.068028 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:03.068000    5678 generic.go:356] "Generic (PLEG): container finished" podID="ee340d2f-4dc4-4d12-9f68-62402f7916b8" containerID="824f8df4d44fcd56baf3df25befeb5b7450e4cefc7540e825aaf1080e074f96c" exitCode=0
Oct 30 21:57:03.068150 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:03.068083    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7lltl" event={"ID":"ee340d2f-4dc4-4d12-9f68-62402f7916b8","Type":"ContainerDied","Data":"824f8df4d44fcd56baf3df25befeb5b7450e4cefc7540e825aaf1080e074f96c"}
Oct 30 21:57:03.070288 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:03.070267    5678 generic.go:356] "Generic (PLEG): container finished" podID="4bfc7de7-025b-4746-847c-7be0eba28b14" containerID="91e0d8cc634f04d438f7feceb4da20bc97c051b3abc398d8e41f46c076be4027" exitCode=0
Oct 30 21:57:03.070373 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:03.070353    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714527g" event={"ID":"4bfc7de7-025b-4746-847c-7be0eba28b14","Type":"ContainerDied","Data":"91e0d8cc634f04d438f7feceb4da20bc97c051b3abc398d8e41f46c076be4027"}
Oct 30 21:57:03.199962 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:03.199936    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644nj4bh"
Oct 30 21:57:03.217839 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:03.217811    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/105ffac7-32e6-415c-bb60-235b2d822169-util\" (UniqueName: \"kubernetes.io/empty-dir/105ffac7-32e6-415c-bb60-235b2d822169-util\") pod \"105ffac7-32e6-415c-bb60-235b2d822169\" (UID: \"105ffac7-32e6-415c-bb60-235b2d822169\") "
Oct 30 21:57:03.217982 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:03.217869    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/105ffac7-32e6-415c-bb60-235b2d822169-bundle\" (UniqueName: \"kubernetes.io/empty-dir/105ffac7-32e6-415c-bb60-235b2d822169-bundle\") pod \"105ffac7-32e6-415c-bb60-235b2d822169\" (UID: \"105ffac7-32e6-415c-bb60-235b2d822169\") "
Oct 30 21:57:03.217982 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:03.217960    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/105ffac7-32e6-415c-bb60-235b2d822169-kube-api-access-pmhnr\" (UniqueName: \"kubernetes.io/projected/105ffac7-32e6-415c-bb60-235b2d822169-kube-api-access-pmhnr\") pod \"105ffac7-32e6-415c-bb60-235b2d822169\" (UID: \"105ffac7-32e6-415c-bb60-235b2d822169\") "
Oct 30 21:57:03.218270 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:03.218243    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/105ffac7-32e6-415c-bb60-235b2d822169-bundle" pod "105ffac7-32e6-415c-bb60-235b2d822169" (UID: "105ffac7-32e6-415c-bb60-235b2d822169"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:57:03.222131 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:03.222102    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/105ffac7-32e6-415c-bb60-235b2d822169-kube-api-access-pmhnr" pod "105ffac7-32e6-415c-bb60-235b2d822169" (UID: "105ffac7-32e6-415c-bb60-235b2d822169"). InnerVolumeSpecName "kube-api-access-pmhnr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:57:03.223844 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:03.223818    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/105ffac7-32e6-415c-bb60-235b2d822169-util" pod "105ffac7-32e6-415c-bb60-235b2d822169" (UID: "105ffac7-32e6-415c-bb60-235b2d822169"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:57:03.244196 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:03.244167    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d788crs"
Oct 30 21:57:03.252441 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:03.252423    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpdjqd"
Oct 30 21:57:03.255388 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:03.255372    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mzkjf"
Oct 30 21:57:03.318540 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:03.318465    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76bac5b5-d258-4942-8a65-b0ef3f55b55b-bundle\" (UniqueName: \"kubernetes.io/empty-dir/76bac5b5-d258-4942-8a65-b0ef3f55b55b-bundle\") pod \"76bac5b5-d258-4942-8a65-b0ef3f55b55b\" (UID: \"76bac5b5-d258-4942-8a65-b0ef3f55b55b\") "
Oct 30 21:57:03.318540 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:03.318517    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8eae4463-6286-4817-9319-bc4b0d528ede-kube-api-access-v549x\" (UniqueName: \"kubernetes.io/projected/8eae4463-6286-4817-9319-bc4b0d528ede-kube-api-access-v549x\") pod \"8eae4463-6286-4817-9319-bc4b0d528ede\" (UID: \"8eae4463-6286-4817-9319-bc4b0d528ede\") "
Oct 30 21:57:03.318540 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:03.318538    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8eae4463-6286-4817-9319-bc4b0d528ede-util\" (UniqueName: \"kubernetes.io/empty-dir/8eae4463-6286-4817-9319-bc4b0d528ede-util\") pod \"8eae4463-6286-4817-9319-bc4b0d528ede\" (UID: \"8eae4463-6286-4817-9319-bc4b0d528ede\") "
Oct 30 21:57:03.318802 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:03.318558    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76bac5b5-d258-4942-8a65-b0ef3f55b55b-util\" (UniqueName: \"kubernetes.io/empty-dir/76bac5b5-d258-4942-8a65-b0ef3f55b55b-util\") pod \"76bac5b5-d258-4942-8a65-b0ef3f55b55b\" (UID: \"76bac5b5-d258-4942-8a65-b0ef3f55b55b\") "
Oct 30 21:57:03.318802 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:03.318601    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1cde967-3ef6-4c48-bef1-59574f7f2479-bundle\" (UniqueName: \"kubernetes.io/empty-dir/d1cde967-3ef6-4c48-bef1-59574f7f2479-bundle\") pod \"d1cde967-3ef6-4c48-bef1-59574f7f2479\" (UID: \"d1cde967-3ef6-4c48-bef1-59574f7f2479\") "
Oct 30 21:57:03.318802 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:03.318620    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8eae4463-6286-4817-9319-bc4b0d528ede-bundle\" (UniqueName: \"kubernetes.io/empty-dir/8eae4463-6286-4817-9319-bc4b0d528ede-bundle\") pod \"8eae4463-6286-4817-9319-bc4b0d528ede\" (UID: \"8eae4463-6286-4817-9319-bc4b0d528ede\") "
Oct 30 21:57:03.318802 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:03.318662    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1cde967-3ef6-4c48-bef1-59574f7f2479-kube-api-access-7p2qw\" (UniqueName: \"kubernetes.io/projected/d1cde967-3ef6-4c48-bef1-59574f7f2479-kube-api-access-7p2qw\") pod \"d1cde967-3ef6-4c48-bef1-59574f7f2479\" (UID: \"d1cde967-3ef6-4c48-bef1-59574f7f2479\") "
Oct 30 21:57:03.318802 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:03.318686    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1cde967-3ef6-4c48-bef1-59574f7f2479-util\" (UniqueName: \"kubernetes.io/empty-dir/d1cde967-3ef6-4c48-bef1-59574f7f2479-util\") pod \"d1cde967-3ef6-4c48-bef1-59574f7f2479\" (UID: \"d1cde967-3ef6-4c48-bef1-59574f7f2479\") "
Oct 30 21:57:03.318802 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:03.318728    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76bac5b5-d258-4942-8a65-b0ef3f55b55b-kube-api-access-9jxgs\" (UniqueName: \"kubernetes.io/projected/76bac5b5-d258-4942-8a65-b0ef3f55b55b-kube-api-access-9jxgs\") pod \"76bac5b5-d258-4942-8a65-b0ef3f55b55b\" (UID: \"76bac5b5-d258-4942-8a65-b0ef3f55b55b\") "
Oct 30 21:57:03.319097 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:03.318960    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmhnr\" (UniqueName: \"kubernetes.io/projected/105ffac7-32e6-415c-bb60-235b2d822169-kube-api-access-pmhnr\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:03.319097 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:03.318980    5678 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/105ffac7-32e6-415c-bb60-235b2d822169-util\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:03.319097 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:03.318996    5678 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/105ffac7-32e6-415c-bb60-235b2d822169-bundle\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:03.319097 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:03.319075    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1cde967-3ef6-4c48-bef1-59574f7f2479-bundle" pod "d1cde967-3ef6-4c48-bef1-59574f7f2479" (UID: "d1cde967-3ef6-4c48-bef1-59574f7f2479"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:57:03.319370 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:03.319333    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76bac5b5-d258-4942-8a65-b0ef3f55b55b-bundle" pod "76bac5b5-d258-4942-8a65-b0ef3f55b55b" (UID: "76bac5b5-d258-4942-8a65-b0ef3f55b55b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:57:03.319721 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:03.319691    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8eae4463-6286-4817-9319-bc4b0d528ede-bundle" pod "8eae4463-6286-4817-9319-bc4b0d528ede" (UID: "8eae4463-6286-4817-9319-bc4b0d528ede"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:57:03.321132 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:03.321108    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76bac5b5-d258-4942-8a65-b0ef3f55b55b-kube-api-access-9jxgs" pod "76bac5b5-d258-4942-8a65-b0ef3f55b55b" (UID: "76bac5b5-d258-4942-8a65-b0ef3f55b55b"). InnerVolumeSpecName "kube-api-access-9jxgs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:57:03.321242 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:03.321108    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8eae4463-6286-4817-9319-bc4b0d528ede-kube-api-access-v549x" pod "8eae4463-6286-4817-9319-bc4b0d528ede" (UID: "8eae4463-6286-4817-9319-bc4b0d528ede"). InnerVolumeSpecName "kube-api-access-v549x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:57:03.321350 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:03.321323    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1cde967-3ef6-4c48-bef1-59574f7f2479-kube-api-access-7p2qw" pod "d1cde967-3ef6-4c48-bef1-59574f7f2479" (UID: "d1cde967-3ef6-4c48-bef1-59574f7f2479"). InnerVolumeSpecName "kube-api-access-7p2qw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:57:03.323558 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:03.323535    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76bac5b5-d258-4942-8a65-b0ef3f55b55b-util" pod "76bac5b5-d258-4942-8a65-b0ef3f55b55b" (UID: "76bac5b5-d258-4942-8a65-b0ef3f55b55b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:57:03.326993 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:03.326970    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8eae4463-6286-4817-9319-bc4b0d528ede-util" pod "8eae4463-6286-4817-9319-bc4b0d528ede" (UID: "8eae4463-6286-4817-9319-bc4b0d528ede"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:57:03.327277 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:03.327263    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1cde967-3ef6-4c48-bef1-59574f7f2479-util" pod "d1cde967-3ef6-4c48-bef1-59574f7f2479" (UID: "d1cde967-3ef6-4c48-bef1-59574f7f2479"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:57:03.420035 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:03.420008    5678 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d1cde967-3ef6-4c48-bef1-59574f7f2479-bundle\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:03.420035 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:03.420031    5678 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8eae4463-6286-4817-9319-bc4b0d528ede-bundle\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:03.420035 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:03.420041    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7p2qw\" (UniqueName: \"kubernetes.io/projected/d1cde967-3ef6-4c48-bef1-59574f7f2479-kube-api-access-7p2qw\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:03.420251 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:03.420051    5678 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d1cde967-3ef6-4c48-bef1-59574f7f2479-util\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:03.420251 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:03.420060    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jxgs\" (UniqueName: \"kubernetes.io/projected/76bac5b5-d258-4942-8a65-b0ef3f55b55b-kube-api-access-9jxgs\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:03.420251 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:03.420068    5678 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/76bac5b5-d258-4942-8a65-b0ef3f55b55b-bundle\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:03.420251 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:03.420077    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v549x\" (UniqueName: \"kubernetes.io/projected/8eae4463-6286-4817-9319-bc4b0d528ede-kube-api-access-v549x\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:03.420251 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:03.420086    5678 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8eae4463-6286-4817-9319-bc4b0d528ede-util\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:03.420251 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:03.420093    5678 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/76bac5b5-d258-4942-8a65-b0ef3f55b55b-util\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:04.075647 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:04.075500    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644nj4bh"
Oct 30 21:57:04.075647 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:04.075527    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644nj4bh" event={"ID":"105ffac7-32e6-415c-bb60-235b2d822169","Type":"ContainerDied","Data":"f4234612ff9d8dd5026435b1a24e678866fc8650a4a2ec297e1570db81b3685d"}
Oct 30 21:57:04.075647 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:04.075575    5678 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f4234612ff9d8dd5026435b1a24e678866fc8650a4a2ec297e1570db81b3685d"
Oct 30 21:57:04.078030 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:04.078008    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d788crs"
Oct 30 21:57:04.078030 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:04.078009    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d788crs" event={"ID":"8eae4463-6286-4817-9319-bc4b0d528ede","Type":"ContainerDied","Data":"c548f77a1149bdcffba8903671d5f68385b565a47f65f3b868c66479c68f62da"}
Oct 30 21:57:04.078225 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:04.078056    5678 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c548f77a1149bdcffba8903671d5f68385b565a47f65f3b868c66479c68f62da"
Oct 30 21:57:04.080086 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:04.080064    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mzkjf"
Oct 30 21:57:04.080086 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:04.080077    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mzkjf" event={"ID":"d1cde967-3ef6-4c48-bef1-59574f7f2479","Type":"ContainerDied","Data":"af2dce08c33df29c5ac63fee7b833b87083b26a47c93f718312e54db74914e9d"}
Oct 30 21:57:04.080299 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:04.080116    5678 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af2dce08c33df29c5ac63fee7b833b87083b26a47c93f718312e54db74914e9d"
Oct 30 21:57:04.082016 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:04.081947    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpdjqd" event={"ID":"76bac5b5-d258-4942-8a65-b0ef3f55b55b","Type":"ContainerDied","Data":"7724e6d8272b8233a9da0bfca53d4127a4b624eba23e388d8870c8b19ec34ccd"}
Oct 30 21:57:04.082016 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:04.081973    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpdjqd"
Oct 30 21:57:04.082016 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:04.081993    5678 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7724e6d8272b8233a9da0bfca53d4127a4b624eba23e388d8870c8b19ec34ccd"
Oct 30 21:57:04.223826 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:04.223798    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csvbt2"
Oct 30 21:57:04.225862 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:04.225838    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2847b05-a3f6-430c-9052-3b73934599f1-util\" (UniqueName: \"kubernetes.io/empty-dir/a2847b05-a3f6-430c-9052-3b73934599f1-util\") pod \"a2847b05-a3f6-430c-9052-3b73934599f1\" (UID: \"a2847b05-a3f6-430c-9052-3b73934599f1\") "
Oct 30 21:57:04.226010 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:04.225894    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2847b05-a3f6-430c-9052-3b73934599f1-bundle\" (UniqueName: \"kubernetes.io/empty-dir/a2847b05-a3f6-430c-9052-3b73934599f1-bundle\") pod \"a2847b05-a3f6-430c-9052-3b73934599f1\" (UID: \"a2847b05-a3f6-430c-9052-3b73934599f1\") "
Oct 30 21:57:04.226010 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:04.225950    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2847b05-a3f6-430c-9052-3b73934599f1-kube-api-access-v5gdj\" (UniqueName: \"kubernetes.io/projected/a2847b05-a3f6-430c-9052-3b73934599f1-kube-api-access-v5gdj\") pod \"a2847b05-a3f6-430c-9052-3b73934599f1\" (UID: \"a2847b05-a3f6-430c-9052-3b73934599f1\") "
Oct 30 21:57:04.226694 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:04.226666    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2847b05-a3f6-430c-9052-3b73934599f1-bundle" pod "a2847b05-a3f6-430c-9052-3b73934599f1" (UID: "a2847b05-a3f6-430c-9052-3b73934599f1"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:57:04.229554 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:04.229528    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2847b05-a3f6-430c-9052-3b73934599f1-kube-api-access-v5gdj" pod "a2847b05-a3f6-430c-9052-3b73934599f1" (UID: "a2847b05-a3f6-430c-9052-3b73934599f1"). InnerVolumeSpecName "kube-api-access-v5gdj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:57:04.232254 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:04.232201    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2847b05-a3f6-430c-9052-3b73934599f1-util" pod "a2847b05-a3f6-430c-9052-3b73934599f1" (UID: "a2847b05-a3f6-430c-9052-3b73934599f1"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:57:04.245749 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:04.245722    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714527g"
Oct 30 21:57:04.253078 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:04.253056    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7lltl"
Oct 30 21:57:04.326599 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:04.326526    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee340d2f-4dc4-4d12-9f68-62402f7916b8-bundle\" (UniqueName: \"kubernetes.io/empty-dir/ee340d2f-4dc4-4d12-9f68-62402f7916b8-bundle\") pod \"ee340d2f-4dc4-4d12-9f68-62402f7916b8\" (UID: \"ee340d2f-4dc4-4d12-9f68-62402f7916b8\") "
Oct 30 21:57:04.326599 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:04.326570    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee340d2f-4dc4-4d12-9f68-62402f7916b8-kube-api-access-qfbvf\" (UniqueName: \"kubernetes.io/projected/ee340d2f-4dc4-4d12-9f68-62402f7916b8-kube-api-access-qfbvf\") pod \"ee340d2f-4dc4-4d12-9f68-62402f7916b8\" (UID: \"ee340d2f-4dc4-4d12-9f68-62402f7916b8\") "
Oct 30 21:57:04.326599 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:04.326591    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bfc7de7-025b-4746-847c-7be0eba28b14-bundle\" (UniqueName: \"kubernetes.io/empty-dir/4bfc7de7-025b-4746-847c-7be0eba28b14-bundle\") pod \"4bfc7de7-025b-4746-847c-7be0eba28b14\" (UID: \"4bfc7de7-025b-4746-847c-7be0eba28b14\") "
Oct 30 21:57:04.326864 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:04.326644    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bfc7de7-025b-4746-847c-7be0eba28b14-kube-api-access-6rpzb\" (UniqueName: \"kubernetes.io/projected/4bfc7de7-025b-4746-847c-7be0eba28b14-kube-api-access-6rpzb\") pod \"4bfc7de7-025b-4746-847c-7be0eba28b14\" (UID: \"4bfc7de7-025b-4746-847c-7be0eba28b14\") "
Oct 30 21:57:04.326864 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:04.326683    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee340d2f-4dc4-4d12-9f68-62402f7916b8-util\" (UniqueName: \"kubernetes.io/empty-dir/ee340d2f-4dc4-4d12-9f68-62402f7916b8-util\") pod \"ee340d2f-4dc4-4d12-9f68-62402f7916b8\" (UID: \"ee340d2f-4dc4-4d12-9f68-62402f7916b8\") "
Oct 30 21:57:04.326864 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:04.326705    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bfc7de7-025b-4746-847c-7be0eba28b14-util\" (UniqueName: \"kubernetes.io/empty-dir/4bfc7de7-025b-4746-847c-7be0eba28b14-util\") pod \"4bfc7de7-025b-4746-847c-7be0eba28b14\" (UID: \"4bfc7de7-025b-4746-847c-7be0eba28b14\") "
Oct 30 21:57:04.326864 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:04.326834    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5gdj\" (UniqueName: \"kubernetes.io/projected/a2847b05-a3f6-430c-9052-3b73934599f1-kube-api-access-v5gdj\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:04.326864 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:04.326845    5678 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a2847b05-a3f6-430c-9052-3b73934599f1-util\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:04.326864 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:04.326854    5678 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a2847b05-a3f6-430c-9052-3b73934599f1-bundle\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:04.327440 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:04.327364    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bfc7de7-025b-4746-847c-7be0eba28b14-bundle" pod "4bfc7de7-025b-4746-847c-7be0eba28b14" (UID: "4bfc7de7-025b-4746-847c-7be0eba28b14"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:57:04.327571 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:04.327542    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee340d2f-4dc4-4d12-9f68-62402f7916b8-bundle" pod "ee340d2f-4dc4-4d12-9f68-62402f7916b8" (UID: "ee340d2f-4dc4-4d12-9f68-62402f7916b8"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:57:04.328880 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:04.328858    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bfc7de7-025b-4746-847c-7be0eba28b14-kube-api-access-6rpzb" pod "4bfc7de7-025b-4746-847c-7be0eba28b14" (UID: "4bfc7de7-025b-4746-847c-7be0eba28b14"). InnerVolumeSpecName "kube-api-access-6rpzb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:57:04.329273 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:04.329219    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee340d2f-4dc4-4d12-9f68-62402f7916b8-kube-api-access-qfbvf" pod "ee340d2f-4dc4-4d12-9f68-62402f7916b8" (UID: "ee340d2f-4dc4-4d12-9f68-62402f7916b8"). InnerVolumeSpecName "kube-api-access-qfbvf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:57:04.331648 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:04.331625    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee340d2f-4dc4-4d12-9f68-62402f7916b8-util" pod "ee340d2f-4dc4-4d12-9f68-62402f7916b8" (UID: "ee340d2f-4dc4-4d12-9f68-62402f7916b8"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:57:04.333352 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:04.333334    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bfc7de7-025b-4746-847c-7be0eba28b14-util" pod "4bfc7de7-025b-4746-847c-7be0eba28b14" (UID: "4bfc7de7-025b-4746-847c-7be0eba28b14"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:57:04.427183 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:04.427157    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rpzb\" (UniqueName: \"kubernetes.io/projected/4bfc7de7-025b-4746-847c-7be0eba28b14-kube-api-access-6rpzb\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:04.427183 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:04.427179    5678 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ee340d2f-4dc4-4d12-9f68-62402f7916b8-util\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:04.427183 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:04.427189    5678 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4bfc7de7-025b-4746-847c-7be0eba28b14-util\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:04.427394 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:04.427197    5678 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ee340d2f-4dc4-4d12-9f68-62402f7916b8-bundle\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:04.427394 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:04.427207    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfbvf\" (UniqueName: \"kubernetes.io/projected/ee340d2f-4dc4-4d12-9f68-62402f7916b8-kube-api-access-qfbvf\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:04.427394 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:04.427215    5678 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4bfc7de7-025b-4746-847c-7be0eba28b14-bundle\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:05.087271 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:05.087236    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7lltl" event={"ID":"ee340d2f-4dc4-4d12-9f68-62402f7916b8","Type":"ContainerDied","Data":"a7be542e6ecb24fedb99473a332dde495aa29db76976947f3785b6707ce94d88"}
Oct 30 21:57:05.087732 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:05.087286    5678 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a7be542e6ecb24fedb99473a332dde495aa29db76976947f3785b6707ce94d88"
Oct 30 21:57:05.087732 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:05.087251    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7lltl"
Oct 30 21:57:05.089371 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:05.089333    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714527g"
Oct 30 21:57:05.089371 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:05.089356    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714527g" event={"ID":"4bfc7de7-025b-4746-847c-7be0eba28b14","Type":"ContainerDied","Data":"7049dcf0b720d8a80d3b98febac7f5224a3e15784ab6df099d2f3894b57257c2"}
Oct 30 21:57:05.089566 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:05.089396    5678 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7049dcf0b720d8a80d3b98febac7f5224a3e15784ab6df099d2f3894b57257c2"
Oct 30 21:57:05.091327 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:05.091303    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csvbt2" event={"ID":"a2847b05-a3f6-430c-9052-3b73934599f1","Type":"ContainerDied","Data":"0a68be6575357ede071e99a5555866c96dfe00ec3f45e604df04b58e714850a9"}
Oct 30 21:57:05.091426 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:05.091346    5678 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0a68be6575357ede071e99a5555866c96dfe00ec3f45e604df04b58e714850a9"
Oct 30 21:57:05.091426 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:05.091358    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csvbt2"
Oct 30 21:57:07.071803 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:07.071771    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-rq5sh"]
Oct 30 21:57:07.072191 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:07.072044    5678 cpu_manager.go:414] "containerMap: removing container" podUID="4bfc7de7-025b-4746-847c-7be0eba28b14" containerName="util"
Oct 30 21:57:07.072191 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:07.072058    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bfc7de7-025b-4746-847c-7be0eba28b14" containerName="util"
Oct 30 21:57:07.072191 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:07.072071    5678 cpu_manager.go:414] "containerMap: removing container" podUID="ee340d2f-4dc4-4d12-9f68-62402f7916b8" containerName="util"
Oct 30 21:57:07.072191 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:07.072076    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee340d2f-4dc4-4d12-9f68-62402f7916b8" containerName="util"
Oct 30 21:57:07.072191 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:07.072084    5678 cpu_manager.go:414] "containerMap: removing container" podUID="d1cde967-3ef6-4c48-bef1-59574f7f2479" containerName="pull"
Oct 30 21:57:07.072191 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:07.072089    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1cde967-3ef6-4c48-bef1-59574f7f2479" containerName="pull"
Oct 30 21:57:07.072191 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:07.072096    5678 cpu_manager.go:414] "containerMap: removing container" podUID="a2847b05-a3f6-430c-9052-3b73934599f1" containerName="extract"
Oct 30 21:57:07.072191 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:07.072101    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2847b05-a3f6-430c-9052-3b73934599f1" containerName="extract"
Oct 30 21:57:07.072191 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:07.072106    5678 cpu_manager.go:414] "containerMap: removing container" podUID="105ffac7-32e6-415c-bb60-235b2d822169" containerName="util"
Oct 30 21:57:07.072191 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:07.072110    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="105ffac7-32e6-415c-bb60-235b2d822169" containerName="util"
Oct 30 21:57:07.072191 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:07.072118    5678 cpu_manager.go:414] "containerMap: removing container" podUID="d1cde967-3ef6-4c48-bef1-59574f7f2479" containerName="extract"
Oct 30 21:57:07.072191 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:07.072122    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1cde967-3ef6-4c48-bef1-59574f7f2479" containerName="extract"
Oct 30 21:57:07.072191 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:07.072137    5678 cpu_manager.go:414] "containerMap: removing container" podUID="8eae4463-6286-4817-9319-bc4b0d528ede" containerName="util"
Oct 30 21:57:07.072191 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:07.072146    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8eae4463-6286-4817-9319-bc4b0d528ede" containerName="util"
Oct 30 21:57:07.072191 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:07.072153    5678 cpu_manager.go:414] "containerMap: removing container" podUID="76bac5b5-d258-4942-8a65-b0ef3f55b55b" containerName="pull"
Oct 30 21:57:07.072191 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:07.072157    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76bac5b5-d258-4942-8a65-b0ef3f55b55b" containerName="pull"
Oct 30 21:57:07.072191 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:07.072163    5678 cpu_manager.go:414] "containerMap: removing container" podUID="105ffac7-32e6-415c-bb60-235b2d822169" containerName="pull"
Oct 30 21:57:07.072191 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:07.072167    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="105ffac7-32e6-415c-bb60-235b2d822169" containerName="pull"
Oct 30 21:57:07.072191 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:07.072183    5678 cpu_manager.go:414] "containerMap: removing container" podUID="a2847b05-a3f6-430c-9052-3b73934599f1" containerName="pull"
Oct 30 21:57:07.072191 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:07.072187    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2847b05-a3f6-430c-9052-3b73934599f1" containerName="pull"
Oct 30 21:57:07.072191 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:07.072194    5678 cpu_manager.go:414] "containerMap: removing container" podUID="8eae4463-6286-4817-9319-bc4b0d528ede" containerName="extract"
Oct 30 21:57:07.072191 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:07.072199    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8eae4463-6286-4817-9319-bc4b0d528ede" containerName="extract"
Oct 30 21:57:07.072827 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:07.072210    5678 cpu_manager.go:414] "containerMap: removing container" podUID="ee340d2f-4dc4-4d12-9f68-62402f7916b8" containerName="extract"
Oct 30 21:57:07.072827 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:07.072218    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee340d2f-4dc4-4d12-9f68-62402f7916b8" containerName="extract"
Oct 30 21:57:07.072827 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:07.072226    5678 cpu_manager.go:414] "containerMap: removing container" podUID="a2847b05-a3f6-430c-9052-3b73934599f1" containerName="util"
Oct 30 21:57:07.072827 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:07.072231    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2847b05-a3f6-430c-9052-3b73934599f1" containerName="util"
Oct 30 21:57:07.072827 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:07.072237    5678 cpu_manager.go:414] "containerMap: removing container" podUID="76bac5b5-d258-4942-8a65-b0ef3f55b55b" containerName="extract"
Oct 30 21:57:07.072827 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:07.072258    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76bac5b5-d258-4942-8a65-b0ef3f55b55b" containerName="extract"
Oct 30 21:57:07.072827 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:07.072266    5678 cpu_manager.go:414] "containerMap: removing container" podUID="76bac5b5-d258-4942-8a65-b0ef3f55b55b" containerName="util"
Oct 30 21:57:07.072827 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:07.072271    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76bac5b5-d258-4942-8a65-b0ef3f55b55b" containerName="util"
Oct 30 21:57:07.072827 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:07.072276    5678 cpu_manager.go:414] "containerMap: removing container" podUID="d1cde967-3ef6-4c48-bef1-59574f7f2479" containerName="util"
Oct 30 21:57:07.072827 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:07.072280    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1cde967-3ef6-4c48-bef1-59574f7f2479" containerName="util"
Oct 30 21:57:07.072827 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:07.072288    5678 cpu_manager.go:414] "containerMap: removing container" podUID="ee340d2f-4dc4-4d12-9f68-62402f7916b8" containerName="pull"
Oct 30 21:57:07.072827 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:07.072294    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee340d2f-4dc4-4d12-9f68-62402f7916b8" containerName="pull"
Oct 30 21:57:07.072827 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:07.072305    5678 cpu_manager.go:414] "containerMap: removing container" podUID="8eae4463-6286-4817-9319-bc4b0d528ede" containerName="pull"
Oct 30 21:57:07.072827 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:07.072309    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8eae4463-6286-4817-9319-bc4b0d528ede" containerName="pull"
Oct 30 21:57:07.072827 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:07.072316    5678 cpu_manager.go:414] "containerMap: removing container" podUID="4bfc7de7-025b-4746-847c-7be0eba28b14" containerName="pull"
Oct 30 21:57:07.072827 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:07.072320    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bfc7de7-025b-4746-847c-7be0eba28b14" containerName="pull"
Oct 30 21:57:07.072827 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:07.072327    5678 cpu_manager.go:414] "containerMap: removing container" podUID="105ffac7-32e6-415c-bb60-235b2d822169" containerName="extract"
Oct 30 21:57:07.072827 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:07.072331    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="105ffac7-32e6-415c-bb60-235b2d822169" containerName="extract"
Oct 30 21:57:07.072827 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:07.072337    5678 cpu_manager.go:414] "containerMap: removing container" podUID="4bfc7de7-025b-4746-847c-7be0eba28b14" containerName="extract"
Oct 30 21:57:07.072827 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:07.072342    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bfc7de7-025b-4746-847c-7be0eba28b14" containerName="extract"
Oct 30 21:57:07.072827 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:07.072387    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="8eae4463-6286-4817-9319-bc4b0d528ede" containerName="extract"
Oct 30 21:57:07.072827 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:07.072395    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee340d2f-4dc4-4d12-9f68-62402f7916b8" containerName="extract"
Oct 30 21:57:07.072827 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:07.072413    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="76bac5b5-d258-4942-8a65-b0ef3f55b55b" containerName="extract"
Oct 30 21:57:07.072827 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:07.072420    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="105ffac7-32e6-415c-bb60-235b2d822169" containerName="extract"
Oct 30 21:57:07.072827 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:07.072425    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bfc7de7-025b-4746-847c-7be0eba28b14" containerName="extract"
Oct 30 21:57:07.072827 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:07.072429    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2847b05-a3f6-430c-9052-3b73934599f1" containerName="extract"
Oct 30 21:57:07.072827 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:07.072434    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1cde967-3ef6-4c48-bef1-59574f7f2479" containerName="extract"
Oct 30 21:57:07.080501 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:07.080478    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-rq5sh"
Oct 30 21:57:07.130206 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:07.130176    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\""
Oct 30 21:57:07.147724 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:07.147695    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-275vv\" (UniqueName: \"kubernetes.io/projected/09e892b0-4d17-412c-87a7-250d34383e0f-kube-api-access-275vv\") pod \"jobset-operator-674cd59dc5-rq5sh\" (UID: \"09e892b0-4d17-412c-87a7-250d34383e0f\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-rq5sh"
Oct 30 21:57:07.147883 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:07.147753    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/09e892b0-4d17-412c-87a7-250d34383e0f-tmp\") pod \"jobset-operator-674cd59dc5-rq5sh\" (UID: \"09e892b0-4d17-412c-87a7-250d34383e0f\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-rq5sh"
Oct 30 21:57:07.177131 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:07.177104    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-pdjp2\""
Oct 30 21:57:07.177243 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:07.177140    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\""
Oct 30 21:57:07.178242 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:07.178222    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-rq5sh"]
Oct 30 21:57:07.248126 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:07.248101    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/09e892b0-4d17-412c-87a7-250d34383e0f-tmp\") pod \"jobset-operator-674cd59dc5-rq5sh\" (UID: \"09e892b0-4d17-412c-87a7-250d34383e0f\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-rq5sh"
Oct 30 21:57:07.248251 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:07.248145    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-275vv\" (UniqueName: \"kubernetes.io/projected/09e892b0-4d17-412c-87a7-250d34383e0f-kube-api-access-275vv\") pod \"jobset-operator-674cd59dc5-rq5sh\" (UID: \"09e892b0-4d17-412c-87a7-250d34383e0f\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-rq5sh"
Oct 30 21:57:07.248500 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:07.248483    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/09e892b0-4d17-412c-87a7-250d34383e0f-tmp\") pod \"jobset-operator-674cd59dc5-rq5sh\" (UID: \"09e892b0-4d17-412c-87a7-250d34383e0f\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-rq5sh"
Oct 30 21:57:07.307023 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:07.306985    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-275vv\" (UniqueName: \"kubernetes.io/projected/09e892b0-4d17-412c-87a7-250d34383e0f-kube-api-access-275vv\") pod \"jobset-operator-674cd59dc5-rq5sh\" (UID: \"09e892b0-4d17-412c-87a7-250d34383e0f\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-rq5sh"
Oct 30 21:57:07.409129 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:07.409102    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-rq5sh"
Oct 30 21:57:07.779137 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:07.779116    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-rq5sh"]
Oct 30 21:57:07.781558 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:57:07.781526    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod09e892b0_4d17_412c_87a7_250d34383e0f.slice/crio-0c591cfab03b1ef1b8d1a4bd7dec31094e5cb8954997465917eb9781c37a5b67 WatchSource:0}: Error finding container 0c591cfab03b1ef1b8d1a4bd7dec31094e5cb8954997465917eb9781c37a5b67: Status 404 returned error can't find the container with id 0c591cfab03b1ef1b8d1a4bd7dec31094e5cb8954997465917eb9781c37a5b67
Oct 30 21:57:08.106027 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:08.105940    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-rq5sh" event={"ID":"09e892b0-4d17-412c-87a7-250d34383e0f","Type":"ContainerStarted","Data":"0c591cfab03b1ef1b8d1a4bd7dec31094e5cb8954997465917eb9781c37a5b67"}
Oct 30 21:57:10.576382 ip-10-0-128-245 kubenswrapper[5678]: E1030 21:57:10.576329    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 21:57:11.118762 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:11.118724    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-rq5sh" event={"ID":"09e892b0-4d17-412c-87a7-250d34383e0f","Type":"ContainerStarted","Data":"d71f65d9548f5754ecc97fe2196e308164282e7368f41ca1cf9a7bf26e5e7726"}
Oct 30 21:57:11.231791 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:11.231631    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-rq5sh" podStartSLOduration=1.083047942 podStartE2EDuration="4.231611366s" totalImagesPullingTime="3.148563424s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:57:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:57:11.230036762 +0000 UTC m=+408.834344316" watchObservedRunningTime="2026-10-30 21:57:11.231611366 +0000 UTC m=+408.835918921"
Oct 30 21:57:14.237482 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:14.237447    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-748948fb58-sg4g4"]
Oct 30 21:57:14.240006 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:14.239990    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-748948fb58-sg4g4"
Oct 30 21:57:14.302535 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:14.302508    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Oct 30 21:57:14.302775 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:14.302741    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Oct 30 21:57:14.302895 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:14.302787    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Oct 30 21:57:14.311170 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:14.311146    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6f9209cb-e7de-4611-9c3a-4f3be4a35b62-apiservice-cert\") pod \"opendatahub-operator-controller-manager-748948fb58-sg4g4\" (UID: \"6f9209cb-e7de-4611-9c3a-4f3be4a35b62\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-748948fb58-sg4g4"
Oct 30 21:57:14.311268 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:14.311185    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6f9209cb-e7de-4611-9c3a-4f3be4a35b62-metrics-cert\") pod \"opendatahub-operator-controller-manager-748948fb58-sg4g4\" (UID: \"6f9209cb-e7de-4611-9c3a-4f3be4a35b62\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-748948fb58-sg4g4"
Oct 30 21:57:14.311268 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:14.311214    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6f9209cb-e7de-4611-9c3a-4f3be4a35b62-manifests\") pod \"opendatahub-operator-controller-manager-748948fb58-sg4g4\" (UID: \"6f9209cb-e7de-4611-9c3a-4f3be4a35b62\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-748948fb58-sg4g4"
Oct 30 21:57:14.311340 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:14.311328    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5z2hp\" (UniqueName: \"kubernetes.io/projected/6f9209cb-e7de-4611-9c3a-4f3be4a35b62-kube-api-access-5z2hp\") pod \"opendatahub-operator-controller-manager-748948fb58-sg4g4\" (UID: \"6f9209cb-e7de-4611-9c3a-4f3be4a35b62\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-748948fb58-sg4g4"
Oct 30 21:57:14.311394 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:14.311380    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6f9209cb-e7de-4611-9c3a-4f3be4a35b62-tmp\") pod \"opendatahub-operator-controller-manager-748948fb58-sg4g4\" (UID: \"6f9209cb-e7de-4611-9c3a-4f3be4a35b62\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-748948fb58-sg4g4"
Oct 30 21:57:14.311456 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:14.311442    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6f9209cb-e7de-4611-9c3a-4f3be4a35b62-webhook-cert\") pod \"opendatahub-operator-controller-manager-748948fb58-sg4g4\" (UID: \"6f9209cb-e7de-4611-9c3a-4f3be4a35b62\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-748948fb58-sg4g4"
Oct 30 21:57:14.341855 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:14.341828    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Oct 30 21:57:14.341997 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:14.341942    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Oct 30 21:57:14.342055 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:14.342023    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-ng7kl\""
Oct 30 21:57:14.343669 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:14.343650    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-748948fb58-sg4g4"]
Oct 30 21:57:14.412207 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:14.412171    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6f9209cb-e7de-4611-9c3a-4f3be4a35b62-manifests\") pod \"opendatahub-operator-controller-manager-748948fb58-sg4g4\" (UID: \"6f9209cb-e7de-4611-9c3a-4f3be4a35b62\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-748948fb58-sg4g4"
Oct 30 21:57:14.412329 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:14.412212    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5z2hp\" (UniqueName: \"kubernetes.io/projected/6f9209cb-e7de-4611-9c3a-4f3be4a35b62-kube-api-access-5z2hp\") pod \"opendatahub-operator-controller-manager-748948fb58-sg4g4\" (UID: \"6f9209cb-e7de-4611-9c3a-4f3be4a35b62\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-748948fb58-sg4g4"
Oct 30 21:57:14.412387 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:14.412344    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6f9209cb-e7de-4611-9c3a-4f3be4a35b62-tmp\") pod \"opendatahub-operator-controller-manager-748948fb58-sg4g4\" (UID: \"6f9209cb-e7de-4611-9c3a-4f3be4a35b62\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-748948fb58-sg4g4"
Oct 30 21:57:14.412452 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:14.412382    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6f9209cb-e7de-4611-9c3a-4f3be4a35b62-webhook-cert\") pod \"opendatahub-operator-controller-manager-748948fb58-sg4g4\" (UID: \"6f9209cb-e7de-4611-9c3a-4f3be4a35b62\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-748948fb58-sg4g4"
Oct 30 21:57:14.412452 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:14.412435    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6f9209cb-e7de-4611-9c3a-4f3be4a35b62-apiservice-cert\") pod \"opendatahub-operator-controller-manager-748948fb58-sg4g4\" (UID: \"6f9209cb-e7de-4611-9c3a-4f3be4a35b62\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-748948fb58-sg4g4"
Oct 30 21:57:14.412536 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:14.412501    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6f9209cb-e7de-4611-9c3a-4f3be4a35b62-metrics-cert\") pod \"opendatahub-operator-controller-manager-748948fb58-sg4g4\" (UID: \"6f9209cb-e7de-4611-9c3a-4f3be4a35b62\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-748948fb58-sg4g4"
Oct 30 21:57:14.412650 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:14.412626    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6f9209cb-e7de-4611-9c3a-4f3be4a35b62-manifests\") pod \"opendatahub-operator-controller-manager-748948fb58-sg4g4\" (UID: \"6f9209cb-e7de-4611-9c3a-4f3be4a35b62\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-748948fb58-sg4g4"
Oct 30 21:57:14.414655 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:14.414626    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6f9209cb-e7de-4611-9c3a-4f3be4a35b62-tmp\") pod \"opendatahub-operator-controller-manager-748948fb58-sg4g4\" (UID: \"6f9209cb-e7de-4611-9c3a-4f3be4a35b62\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-748948fb58-sg4g4"
Oct 30 21:57:14.414771 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:14.414690    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6f9209cb-e7de-4611-9c3a-4f3be4a35b62-webhook-cert\") pod \"opendatahub-operator-controller-manager-748948fb58-sg4g4\" (UID: \"6f9209cb-e7de-4611-9c3a-4f3be4a35b62\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-748948fb58-sg4g4"
Oct 30 21:57:14.414833 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:14.414786    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6f9209cb-e7de-4611-9c3a-4f3be4a35b62-apiservice-cert\") pod \"opendatahub-operator-controller-manager-748948fb58-sg4g4\" (UID: \"6f9209cb-e7de-4611-9c3a-4f3be4a35b62\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-748948fb58-sg4g4"
Oct 30 21:57:14.414873 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:14.414847    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6f9209cb-e7de-4611-9c3a-4f3be4a35b62-metrics-cert\") pod \"opendatahub-operator-controller-manager-748948fb58-sg4g4\" (UID: \"6f9209cb-e7de-4611-9c3a-4f3be4a35b62\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-748948fb58-sg4g4"
Oct 30 21:57:14.470792 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:14.470765    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5z2hp\" (UniqueName: \"kubernetes.io/projected/6f9209cb-e7de-4611-9c3a-4f3be4a35b62-kube-api-access-5z2hp\") pod \"opendatahub-operator-controller-manager-748948fb58-sg4g4\" (UID: \"6f9209cb-e7de-4611-9c3a-4f3be4a35b62\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-748948fb58-sg4g4"
Oct 30 21:57:14.552998 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:14.552931    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-748948fb58-sg4g4"
Oct 30 21:57:15.044317 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:15.044288    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-748948fb58-sg4g4"]
Oct 30 21:57:15.048570 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:57:15.048536    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f9209cb_e7de_4611_9c3a_4f3be4a35b62.slice/crio-38c6fe0a32627da1bb976030fa1b6715c156913e7257f4e64be69321d9923ddd WatchSource:0}: Error finding container 38c6fe0a32627da1bb976030fa1b6715c156913e7257f4e64be69321d9923ddd: Status 404 returned error can't find the container with id 38c6fe0a32627da1bb976030fa1b6715c156913e7257f4e64be69321d9923ddd
Oct 30 21:57:15.133831 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:15.133796    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-748948fb58-sg4g4" event={"ID":"6f9209cb-e7de-4611-9c3a-4f3be4a35b62","Type":"ContainerStarted","Data":"38c6fe0a32627da1bb976030fa1b6715c156913e7257f4e64be69321d9923ddd"}
Oct 30 21:57:19.154548 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:19.154515    5678 generic.go:356] "Generic (PLEG): container finished" podID="6f9209cb-e7de-4611-9c3a-4f3be4a35b62" containerID="ddbc8947282f263aabfea0740dfcd448d0ca3d1d3db19349f3fa38175b17166b" exitCode=0
Oct 30 21:57:19.154879 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:19.154590    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-748948fb58-sg4g4" event={"ID":"6f9209cb-e7de-4611-9c3a-4f3be4a35b62","Type":"ContainerDied","Data":"ddbc8947282f263aabfea0740dfcd448d0ca3d1d3db19349f3fa38175b17166b"}
Oct 30 21:57:20.160862 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:20.160828    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-748948fb58-sg4g4" event={"ID":"6f9209cb-e7de-4611-9c3a-4f3be4a35b62","Type":"ContainerStarted","Data":"bc10fca7358e386e3a0dfee6cd721e7253fea21ceece115165360a9b59d08bd5"}
Oct 30 21:57:20.161328 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:20.160969    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-748948fb58-sg4g4"
Oct 30 21:57:20.355576 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:20.355480    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-748948fb58-sg4g4" podStartSLOduration=1.452621939 podStartE2EDuration="6.355467411s" totalImagesPullingTime="4.453815252s" totalInitContainerRuntime="449.03022ms" isStatefulPod=false podCreationTimestamp="2026-10-30 21:57:14 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:57:20.353608641 +0000 UTC m=+417.957916196" watchObservedRunningTime="2026-10-30 21:57:20.355467411 +0000 UTC m=+417.959774964"
Oct 30 21:57:29.976379 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:29.976334    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-446sc"]
Oct 30 21:57:29.981184 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:29.981164    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-446sc"
Oct 30 21:57:30.036532 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:30.036505    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 30 21:57:30.044878 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:30.044852    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gm9fw\" (UniqueName: \"kubernetes.io/projected/fcbb7898-1962-48be-83bb-0bc88f34674d-kube-api-access-gm9fw\") pod \"dns-operator-controller-manager-8557cbdb84-446sc\" (UID: \"fcbb7898-1962-48be-83bb-0bc88f34674d\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-446sc"
Oct 30 21:57:30.082851 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:30.082826    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Oct 30 21:57:30.082986 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:30.082961    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 30 21:57:30.083209 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:30.083186    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-v8ch2\""
Oct 30 21:57:30.084329 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:30.084308    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-446sc"]
Oct 30 21:57:30.146209 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:30.146179    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gm9fw\" (UniqueName: \"kubernetes.io/projected/fcbb7898-1962-48be-83bb-0bc88f34674d-kube-api-access-gm9fw\") pod \"dns-operator-controller-manager-8557cbdb84-446sc\" (UID: \"fcbb7898-1962-48be-83bb-0bc88f34674d\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-446sc"
Oct 30 21:57:30.214363 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:30.214331    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gm9fw\" (UniqueName: \"kubernetes.io/projected/fcbb7898-1962-48be-83bb-0bc88f34674d-kube-api-access-gm9fw\") pod \"dns-operator-controller-manager-8557cbdb84-446sc\" (UID: \"fcbb7898-1962-48be-83bb-0bc88f34674d\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-446sc"
Oct 30 21:57:30.292788 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:30.292717    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-446sc"
Oct 30 21:57:30.676746 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:30.676725    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-446sc"]
Oct 30 21:57:30.678798 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:57:30.678768    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcbb7898_1962_48be_83bb_0bc88f34674d.slice/crio-6e619556a76a2de775a9d60eab28fe7b89ba5d6de6afb8cb8318435f7356c5db WatchSource:0}: Error finding container 6e619556a76a2de775a9d60eab28fe7b89ba5d6de6afb8cb8318435f7356c5db: Status 404 returned error can't find the container with id 6e619556a76a2de775a9d60eab28fe7b89ba5d6de6afb8cb8318435f7356c5db
Oct 30 21:57:31.167843 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:31.167814    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-748948fb58-sg4g4"
Oct 30 21:57:31.207977 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:31.207939    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-446sc" event={"ID":"fcbb7898-1962-48be-83bb-0bc88f34674d","Type":"ContainerStarted","Data":"6e619556a76a2de775a9d60eab28fe7b89ba5d6de6afb8cb8318435f7356c5db"}
Oct 30 21:57:34.578391 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:34.578358    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-w2tw2"]
Oct 30 21:57:34.580909 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:34.580884    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-w2tw2"
Oct 30 21:57:34.637137 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:34.637101    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-4x9d4\""
Oct 30 21:57:34.681589 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:34.681561    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-w2tw2"]
Oct 30 21:57:34.684428 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:34.684252    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2qps\" (UniqueName: \"kubernetes.io/projected/fb169f7f-ab9f-4619-989c-20b7c5c150e9-kube-api-access-r2qps\") pod \"authorino-operator-7b7bb4bc77-w2tw2\" (UID: \"fb169f7f-ab9f-4619-989c-20b7c5c150e9\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-w2tw2"
Oct 30 21:57:34.785645 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:34.785608    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2qps\" (UniqueName: \"kubernetes.io/projected/fb169f7f-ab9f-4619-989c-20b7c5c150e9-kube-api-access-r2qps\") pod \"authorino-operator-7b7bb4bc77-w2tw2\" (UID: \"fb169f7f-ab9f-4619-989c-20b7c5c150e9\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-w2tw2"
Oct 30 21:57:34.850333 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:34.850257    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2qps\" (UniqueName: \"kubernetes.io/projected/fb169f7f-ab9f-4619-989c-20b7c5c150e9-kube-api-access-r2qps\") pod \"authorino-operator-7b7bb4bc77-w2tw2\" (UID: \"fb169f7f-ab9f-4619-989c-20b7c5c150e9\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-w2tw2"
Oct 30 21:57:34.895501 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:34.895479    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-w2tw2"
Oct 30 21:57:35.225699 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:35.225653    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-446sc" event={"ID":"fcbb7898-1962-48be-83bb-0bc88f34674d","Type":"ContainerStarted","Data":"f1077e7a9862c0f957ceeab7f605470d6bb7867ebfc0ababce7aaec96155c835"}
Oct 30 21:57:35.225870 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:35.225826    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-446sc"
Oct 30 21:57:35.289054 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:35.288960    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-w2tw2"]
Oct 30 21:57:35.291048 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:57:35.291022    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb169f7f_ab9f_4619_989c_20b7c5c150e9.slice/crio-f6f19bb1a589413a96d1338999e7f6ed3f56c17e527e08ad02dc7ee6b102f887 WatchSource:0}: Error finding container f6f19bb1a589413a96d1338999e7f6ed3f56c17e527e08ad02dc7ee6b102f887: Status 404 returned error can't find the container with id f6f19bb1a589413a96d1338999e7f6ed3f56c17e527e08ad02dc7ee6b102f887
Oct 30 21:57:35.342049 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:35.342003    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-446sc" podStartSLOduration=2.160946084 podStartE2EDuration="6.341989088s" totalImagesPullingTime="4.181043004s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:57:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:57:35.340113478 +0000 UTC m=+432.944421031" watchObservedRunningTime="2026-10-30 21:57:35.341989088 +0000 UTC m=+432.946296639"
Oct 30 21:57:35.409911 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:35.409875    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-jjlvw"]
Oct 30 21:57:35.412413 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:35.412377    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-jjlvw"
Oct 30 21:57:35.463847 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:35.463821    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-vsjq6\""
Oct 30 21:57:35.467084 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:35.467062    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Oct 30 21:57:35.467236 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:35.467217    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Oct 30 21:57:35.474762 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:35.474744    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-jjlvw"]
Oct 30 21:57:35.493704 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:35.493648    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrxm8\" (UniqueName: \"kubernetes.io/projected/67f802f4-805b-4503-88f7-31a0da8e6172-kube-api-access-lrxm8\") pod \"cert-manager-webhook-5dbbbf75cc-jjlvw\" (UID: \"67f802f4-805b-4503-88f7-31a0da8e6172\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-jjlvw"
Oct 30 21:57:35.493704 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:35.493689    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/67f802f4-805b-4503-88f7-31a0da8e6172-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-jjlvw\" (UID: \"67f802f4-805b-4503-88f7-31a0da8e6172\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-jjlvw"
Oct 30 21:57:35.594360 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:35.594330    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrxm8\" (UniqueName: \"kubernetes.io/projected/67f802f4-805b-4503-88f7-31a0da8e6172-kube-api-access-lrxm8\") pod \"cert-manager-webhook-5dbbbf75cc-jjlvw\" (UID: \"67f802f4-805b-4503-88f7-31a0da8e6172\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-jjlvw"
Oct 30 21:57:35.594778 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:35.594378    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/67f802f4-805b-4503-88f7-31a0da8e6172-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-jjlvw\" (UID: \"67f802f4-805b-4503-88f7-31a0da8e6172\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-jjlvw"
Oct 30 21:57:35.665579 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:35.665552    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrxm8\" (UniqueName: \"kubernetes.io/projected/67f802f4-805b-4503-88f7-31a0da8e6172-kube-api-access-lrxm8\") pod \"cert-manager-webhook-5dbbbf75cc-jjlvw\" (UID: \"67f802f4-805b-4503-88f7-31a0da8e6172\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-jjlvw"
Oct 30 21:57:35.669482 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:35.669453    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/67f802f4-805b-4503-88f7-31a0da8e6172-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-jjlvw\" (UID: \"67f802f4-805b-4503-88f7-31a0da8e6172\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-jjlvw"
Oct 30 21:57:35.723541 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:35.723515    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-jjlvw"
Oct 30 21:57:36.112062 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:36.112039    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-jjlvw"]
Oct 30 21:57:36.113234 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:57:36.113197    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod67f802f4_805b_4503_88f7_31a0da8e6172.slice/crio-146fd37338b2bddff7bafc326bcef9ca5472594b0a0bca823cddf153fcffb0c9 WatchSource:0}: Error finding container 146fd37338b2bddff7bafc326bcef9ca5472594b0a0bca823cddf153fcffb0c9: Status 404 returned error can't find the container with id 146fd37338b2bddff7bafc326bcef9ca5472594b0a0bca823cddf153fcffb0c9
Oct 30 21:57:36.231711 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:36.231674    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-w2tw2" event={"ID":"fb169f7f-ab9f-4619-989c-20b7c5c150e9","Type":"ContainerStarted","Data":"f6f19bb1a589413a96d1338999e7f6ed3f56c17e527e08ad02dc7ee6b102f887"}
Oct 30 21:57:36.232843 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:36.232816    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-jjlvw" event={"ID":"67f802f4-805b-4503-88f7-31a0da8e6172","Type":"ContainerStarted","Data":"146fd37338b2bddff7bafc326bcef9ca5472594b0a0bca823cddf153fcffb0c9"}
Oct 30 21:57:37.151070 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:37.151031    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-hjfxh"]
Oct 30 21:57:37.154643 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:37.154619    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-hjfxh"
Oct 30 21:57:37.204906 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:37.204880    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-5gmhw\""
Oct 30 21:57:37.210039 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:37.210015    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-hjfxh"]
Oct 30 21:57:37.308540 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:37.308506    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/40cedb52-6b7b-411e-9b2e-7a9572486af6-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-hjfxh\" (UID: \"40cedb52-6b7b-411e-9b2e-7a9572486af6\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-hjfxh"
Oct 30 21:57:37.308540 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:37.308539    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6j96\" (UniqueName: \"kubernetes.io/projected/40cedb52-6b7b-411e-9b2e-7a9572486af6-kube-api-access-k6j96\") pod \"cert-manager-cainjector-5c57bbd4fc-hjfxh\" (UID: \"40cedb52-6b7b-411e-9b2e-7a9572486af6\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-hjfxh"
Oct 30 21:57:37.409374 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:37.409279    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/40cedb52-6b7b-411e-9b2e-7a9572486af6-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-hjfxh\" (UID: \"40cedb52-6b7b-411e-9b2e-7a9572486af6\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-hjfxh"
Oct 30 21:57:37.409374 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:37.409320    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6j96\" (UniqueName: \"kubernetes.io/projected/40cedb52-6b7b-411e-9b2e-7a9572486af6-kube-api-access-k6j96\") pod \"cert-manager-cainjector-5c57bbd4fc-hjfxh\" (UID: \"40cedb52-6b7b-411e-9b2e-7a9572486af6\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-hjfxh"
Oct 30 21:57:37.474354 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:37.474321    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/40cedb52-6b7b-411e-9b2e-7a9572486af6-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-hjfxh\" (UID: \"40cedb52-6b7b-411e-9b2e-7a9572486af6\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-hjfxh"
Oct 30 21:57:37.475255 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:37.475234    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6j96\" (UniqueName: \"kubernetes.io/projected/40cedb52-6b7b-411e-9b2e-7a9572486af6-kube-api-access-k6j96\") pod \"cert-manager-cainjector-5c57bbd4fc-hjfxh\" (UID: \"40cedb52-6b7b-411e-9b2e-7a9572486af6\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-hjfxh"
Oct 30 21:57:37.765509 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:37.765433    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-hjfxh"
Oct 30 21:57:37.920045 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:37.920014    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-vdfd2"]
Oct 30 21:57:37.922739 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:37.922714    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vdfd2"
Oct 30 21:57:37.982757 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:37.982724    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Oct 30 21:57:37.982947 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:37.982828    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-7m7zw\""
Oct 30 21:57:37.982947 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:37.982915    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Oct 30 21:57:38.015282 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:38.015244    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/257a2b42-2326-44b2-840b-a2e974882d20-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-vdfd2\" (UID: \"257a2b42-2326-44b2-840b-a2e974882d20\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vdfd2"
Oct 30 21:57:38.015490 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:38.015296    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/257a2b42-2326-44b2-840b-a2e974882d20-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-vdfd2\" (UID: \"257a2b42-2326-44b2-840b-a2e974882d20\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vdfd2"
Oct 30 21:57:38.015490 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:38.015327    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gd6qm\" (UniqueName: \"kubernetes.io/projected/257a2b42-2326-44b2-840b-a2e974882d20-kube-api-access-gd6qm\") pod \"kuadrant-console-plugin-66c4fb575d-vdfd2\" (UID: \"257a2b42-2326-44b2-840b-a2e974882d20\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vdfd2"
Oct 30 21:57:38.027481 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:38.027376    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-vdfd2"]
Oct 30 21:57:38.117415 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:38.116230    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/257a2b42-2326-44b2-840b-a2e974882d20-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-vdfd2\" (UID: \"257a2b42-2326-44b2-840b-a2e974882d20\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vdfd2"
Oct 30 21:57:38.117415 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:38.116306    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/257a2b42-2326-44b2-840b-a2e974882d20-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-vdfd2\" (UID: \"257a2b42-2326-44b2-840b-a2e974882d20\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vdfd2"
Oct 30 21:57:38.117415 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:38.116340    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gd6qm\" (UniqueName: \"kubernetes.io/projected/257a2b42-2326-44b2-840b-a2e974882d20-kube-api-access-gd6qm\") pod \"kuadrant-console-plugin-66c4fb575d-vdfd2\" (UID: \"257a2b42-2326-44b2-840b-a2e974882d20\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vdfd2"
Oct 30 21:57:38.117415 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:38.117348    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/257a2b42-2326-44b2-840b-a2e974882d20-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-vdfd2\" (UID: \"257a2b42-2326-44b2-840b-a2e974882d20\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vdfd2"
Oct 30 21:57:38.119716 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:38.119686    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/257a2b42-2326-44b2-840b-a2e974882d20-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-vdfd2\" (UID: \"257a2b42-2326-44b2-840b-a2e974882d20\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vdfd2"
Oct 30 21:57:38.157794 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:38.157659    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-hjfxh"]
Oct 30 21:57:38.161464 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:57:38.161429    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod40cedb52_6b7b_411e_9b2e_7a9572486af6.slice/crio-cccd498914e65d3c2f2d2ff1ad2d6163c238f56c7f4a7d485cc4e62b12a838c5 WatchSource:0}: Error finding container cccd498914e65d3c2f2d2ff1ad2d6163c238f56c7f4a7d485cc4e62b12a838c5: Status 404 returned error can't find the container with id cccd498914e65d3c2f2d2ff1ad2d6163c238f56c7f4a7d485cc4e62b12a838c5
Oct 30 21:57:38.175372 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:38.175347    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gd6qm\" (UniqueName: \"kubernetes.io/projected/257a2b42-2326-44b2-840b-a2e974882d20-kube-api-access-gd6qm\") pod \"kuadrant-console-plugin-66c4fb575d-vdfd2\" (UID: \"257a2b42-2326-44b2-840b-a2e974882d20\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vdfd2"
Oct 30 21:57:38.239839 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:38.238935    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vdfd2"
Oct 30 21:57:38.244483 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:38.244435    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-hjfxh" event={"ID":"40cedb52-6b7b-411e-9b2e-7a9572486af6","Type":"ContainerStarted","Data":"cccd498914e65d3c2f2d2ff1ad2d6163c238f56c7f4a7d485cc4e62b12a838c5"}
Oct 30 21:57:38.628427 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:38.628374    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-vdfd2"]
Oct 30 21:57:38.632136 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:57:38.632104    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod257a2b42_2326_44b2_840b_a2e974882d20.slice/crio-c5278f62a0e2303410a8e1774d67b5252efbb1e2c0364127d3ac4aaac115db9a WatchSource:0}: Error finding container c5278f62a0e2303410a8e1774d67b5252efbb1e2c0364127d3ac4aaac115db9a: Status 404 returned error can't find the container with id c5278f62a0e2303410a8e1774d67b5252efbb1e2c0364127d3ac4aaac115db9a
Oct 30 21:57:39.251023 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:39.250989    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vdfd2" event={"ID":"257a2b42-2326-44b2-840b-a2e974882d20","Type":"ContainerStarted","Data":"c5278f62a0e2303410a8e1774d67b5252efbb1e2c0364127d3ac4aaac115db9a"}
Oct 30 21:57:39.252966 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:39.252930    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-w2tw2" event={"ID":"fb169f7f-ab9f-4619-989c-20b7c5c150e9","Type":"ContainerStarted","Data":"fe2b5b20b17552638807ef701097aa25ef5fa7f1c83770e3fe18acffc2fc10e5"}
Oct 30 21:57:39.253155 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:39.253103    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-w2tw2"
Oct 30 21:57:39.367358 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:39.367310    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-w2tw2" podStartSLOduration=2.321283124 podStartE2EDuration="5.36729777s" totalImagesPullingTime="3.046014646s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:57:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:57:39.366753624 +0000 UTC m=+436.971061177" watchObservedRunningTime="2026-10-30 21:57:39.36729777 +0000 UTC m=+436.971605320"
Oct 30 21:57:40.258524 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:40.258421    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-jjlvw" event={"ID":"67f802f4-805b-4503-88f7-31a0da8e6172","Type":"ContainerStarted","Data":"302b7b46044123c02c9513662e264d16a80f2069dd6219b1f1e48e793fc26d6e"}
Oct 30 21:57:40.258524 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:40.258477    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-jjlvw"
Oct 30 21:57:40.260030 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:40.260001    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-hjfxh" event={"ID":"40cedb52-6b7b-411e-9b2e-7a9572486af6","Type":"ContainerStarted","Data":"4f759dbb37df2f78406a2da0ad800e5efef4dd14cd3447a5f803cd2bf50b7248"}
Oct 30 21:57:40.375613 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:40.375565    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-jjlvw" podStartSLOduration=1.499436375 podStartE2EDuration="5.375551703s" totalImagesPullingTime="3.876115328s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:57:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:57:40.373848566 +0000 UTC m=+437.978156129" watchObservedRunningTime="2026-10-30 21:57:40.375551703 +0000 UTC m=+437.979859255"
Oct 30 21:57:40.505978 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:40.505910    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-hjfxh" podStartSLOduration=1.591361926 podStartE2EDuration="3.505895709s" totalImagesPullingTime="1.914533783s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:57:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:57:40.505116198 +0000 UTC m=+438.109423751" watchObservedRunningTime="2026-10-30 21:57:40.505895709 +0000 UTC m=+438.110203258"
Oct 30 21:57:40.572287 ip-10-0-128-245 kubenswrapper[5678]: E1030 21:57:40.572199    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 21:57:45.298504 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:45.298463    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vdfd2" event={"ID":"257a2b42-2326-44b2-840b-a2e974882d20","Type":"ContainerStarted","Data":"3bfe5521eab1d25ad5f16713eb37240a651d3c8255337c7f9244a16142935be2"}
Oct 30 21:57:45.444053 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:45.444007    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vdfd2" podStartSLOduration=2.171713661 podStartE2EDuration="8.443992866s" totalImagesPullingTime="6.272279205s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:57:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:57:45.435075905 +0000 UTC m=+443.039383458" watchObservedRunningTime="2026-10-30 21:57:45.443992866 +0000 UTC m=+443.048300418"
Oct 30 21:57:46.235461 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:46.235428    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-446sc"
Oct 30 21:57:46.269065 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:46.269036    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-jjlvw"
Oct 30 21:57:49.436817 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:49.436781    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-84877cfc-5ktml"]
Oct 30 21:57:49.442038 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:49.442019    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-84877cfc-5ktml"
Oct 30 21:57:49.503662 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:49.503632    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 30 21:57:49.503829 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:49.503685    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 30 21:57:49.538598 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:49.538574    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Oct 30 21:57:49.538766 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:49.538690    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Oct 30 21:57:49.538993 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:49.538974    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-hnqxt\""
Oct 30 21:57:49.540224 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:49.540199    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-84877cfc-5ktml"]
Oct 30 21:57:49.620415 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:49.620367    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8e83b65-8951-43d2-8943-0a0bce44b40c-tls-certs\") pod \"kube-auth-proxy-84877cfc-5ktml\" (UID: \"b8e83b65-8951-43d2-8943-0a0bce44b40c\") " pod="openshift-ingress/kube-auth-proxy-84877cfc-5ktml"
Oct 30 21:57:49.620567 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:49.620496    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xbkx\" (UniqueName: \"kubernetes.io/projected/b8e83b65-8951-43d2-8943-0a0bce44b40c-kube-api-access-2xbkx\") pod \"kube-auth-proxy-84877cfc-5ktml\" (UID: \"b8e83b65-8951-43d2-8943-0a0bce44b40c\") " pod="openshift-ingress/kube-auth-proxy-84877cfc-5ktml"
Oct 30 21:57:49.620567 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:49.620551    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b8e83b65-8951-43d2-8943-0a0bce44b40c-tmp\") pod \"kube-auth-proxy-84877cfc-5ktml\" (UID: \"b8e83b65-8951-43d2-8943-0a0bce44b40c\") " pod="openshift-ingress/kube-auth-proxy-84877cfc-5ktml"
Oct 30 21:57:49.721737 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:49.721658    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xbkx\" (UniqueName: \"kubernetes.io/projected/b8e83b65-8951-43d2-8943-0a0bce44b40c-kube-api-access-2xbkx\") pod \"kube-auth-proxy-84877cfc-5ktml\" (UID: \"b8e83b65-8951-43d2-8943-0a0bce44b40c\") " pod="openshift-ingress/kube-auth-proxy-84877cfc-5ktml"
Oct 30 21:57:49.721737 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:49.721714    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b8e83b65-8951-43d2-8943-0a0bce44b40c-tmp\") pod \"kube-auth-proxy-84877cfc-5ktml\" (UID: \"b8e83b65-8951-43d2-8943-0a0bce44b40c\") " pod="openshift-ingress/kube-auth-proxy-84877cfc-5ktml"
Oct 30 21:57:49.721737 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:49.721734    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8e83b65-8951-43d2-8943-0a0bce44b40c-tls-certs\") pod \"kube-auth-proxy-84877cfc-5ktml\" (UID: \"b8e83b65-8951-43d2-8943-0a0bce44b40c\") " pod="openshift-ingress/kube-auth-proxy-84877cfc-5ktml"
Oct 30 21:57:49.723924 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:49.723896    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b8e83b65-8951-43d2-8943-0a0bce44b40c-tmp\") pod \"kube-auth-proxy-84877cfc-5ktml\" (UID: \"b8e83b65-8951-43d2-8943-0a0bce44b40c\") " pod="openshift-ingress/kube-auth-proxy-84877cfc-5ktml"
Oct 30 21:57:49.724246 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:49.724226    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8e83b65-8951-43d2-8943-0a0bce44b40c-tls-certs\") pod \"kube-auth-proxy-84877cfc-5ktml\" (UID: \"b8e83b65-8951-43d2-8943-0a0bce44b40c\") " pod="openshift-ingress/kube-auth-proxy-84877cfc-5ktml"
Oct 30 21:57:49.785891 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:49.785860    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xbkx\" (UniqueName: \"kubernetes.io/projected/b8e83b65-8951-43d2-8943-0a0bce44b40c-kube-api-access-2xbkx\") pod \"kube-auth-proxy-84877cfc-5ktml\" (UID: \"b8e83b65-8951-43d2-8943-0a0bce44b40c\") " pod="openshift-ingress/kube-auth-proxy-84877cfc-5ktml"
Oct 30 21:57:50.053906 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:50.053820    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-84877cfc-5ktml"
Oct 30 21:57:50.262284 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:50.262256    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-w2tw2"
Oct 30 21:57:50.437773 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:50.437741    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-84877cfc-5ktml"]
Oct 30 21:57:50.472501 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:57:50.472471    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8e83b65_8951_43d2_8943_0a0bce44b40c.slice/crio-ab23b57159e931aea1be94d22ecb4c8cd3ce77584148b570307cb42a85975121 WatchSource:0}: Error finding container ab23b57159e931aea1be94d22ecb4c8cd3ce77584148b570307cb42a85975121: Status 404 returned error can't find the container with id ab23b57159e931aea1be94d22ecb4c8cd3ce77584148b570307cb42a85975121
Oct 30 21:57:51.325607 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:51.325568    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-84877cfc-5ktml" event={"ID":"b8e83b65-8951-43d2-8943-0a0bce44b40c","Type":"ContainerStarted","Data":"ab23b57159e931aea1be94d22ecb4c8cd3ce77584148b570307cb42a85975121"}
Oct 30 21:57:54.341095 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:54.341056    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-84877cfc-5ktml" event={"ID":"b8e83b65-8951-43d2-8943-0a0bce44b40c","Type":"ContainerStarted","Data":"818f1d2ddbd6300c305c9ba64ff0a2059f3e153b11189c5ab953484b3fa585c9"}
Oct 30 21:57:54.485251 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:57:54.485202    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-84877cfc-5ktml" podStartSLOduration=2.616042824 podStartE2EDuration="5.485184781s" totalImagesPullingTime="2.869141957s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:57:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:57:54.484477319 +0000 UTC m=+452.088784871" watchObservedRunningTime="2026-10-30 21:57:54.485184781 +0000 UTC m=+452.089492335"
Oct 30 21:58:00.894613 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:58:00.894576    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-65484b555f-jrtff"]
Oct 30 21:58:00.899324 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:58:00.899299    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-65484b555f-jrtff"
Oct 30 21:58:00.995733 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:58:00.995701    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\""
Oct 30 21:58:00.995733 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:58:00.995723    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\""
Oct 30 21:58:00.995957 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:58:00.995845    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\""
Oct 30 21:58:00.997292 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:58:00.997269    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-65484b555f-jrtff"]
Oct 30 21:58:01.023626 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:58:01.023602    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjt99\" (UniqueName: \"kubernetes.io/projected/20becfa8-0f5d-490d-bf9f-2be53070a1b9-kube-api-access-pjt99\") pod \"jobset-controller-manager-65484b555f-jrtff\" (UID: \"20becfa8-0f5d-490d-bf9f-2be53070a1b9\") " pod="openshift-jobset-operator/jobset-controller-manager-65484b555f-jrtff"
Oct 30 21:58:01.023759 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:58:01.023661    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/20becfa8-0f5d-490d-bf9f-2be53070a1b9-manager-config\") pod \"jobset-controller-manager-65484b555f-jrtff\" (UID: \"20becfa8-0f5d-490d-bf9f-2be53070a1b9\") " pod="openshift-jobset-operator/jobset-controller-manager-65484b555f-jrtff"
Oct 30 21:58:01.023801 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:58:01.023759    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/20becfa8-0f5d-490d-bf9f-2be53070a1b9-cert\") pod \"jobset-controller-manager-65484b555f-jrtff\" (UID: \"20becfa8-0f5d-490d-bf9f-2be53070a1b9\") " pod="openshift-jobset-operator/jobset-controller-manager-65484b555f-jrtff"
Oct 30 21:58:01.023801 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:58:01.023795    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20becfa8-0f5d-490d-bf9f-2be53070a1b9-metrics-certs\") pod \"jobset-controller-manager-65484b555f-jrtff\" (UID: \"20becfa8-0f5d-490d-bf9f-2be53070a1b9\") " pod="openshift-jobset-operator/jobset-controller-manager-65484b555f-jrtff"
Oct 30 21:58:01.049544 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:58:01.049520    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-pmdzg\""
Oct 30 21:58:01.125107 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:58:01.125035    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/20becfa8-0f5d-490d-bf9f-2be53070a1b9-cert\") pod \"jobset-controller-manager-65484b555f-jrtff\" (UID: \"20becfa8-0f5d-490d-bf9f-2be53070a1b9\") " pod="openshift-jobset-operator/jobset-controller-manager-65484b555f-jrtff"
Oct 30 21:58:01.125107 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:58:01.125073    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20becfa8-0f5d-490d-bf9f-2be53070a1b9-metrics-certs\") pod \"jobset-controller-manager-65484b555f-jrtff\" (UID: \"20becfa8-0f5d-490d-bf9f-2be53070a1b9\") " pod="openshift-jobset-operator/jobset-controller-manager-65484b555f-jrtff"
Oct 30 21:58:01.125107 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:58:01.125101    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjt99\" (UniqueName: \"kubernetes.io/projected/20becfa8-0f5d-490d-bf9f-2be53070a1b9-kube-api-access-pjt99\") pod \"jobset-controller-manager-65484b555f-jrtff\" (UID: \"20becfa8-0f5d-490d-bf9f-2be53070a1b9\") " pod="openshift-jobset-operator/jobset-controller-manager-65484b555f-jrtff"
Oct 30 21:58:01.125391 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:58:01.125144    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/20becfa8-0f5d-490d-bf9f-2be53070a1b9-manager-config\") pod \"jobset-controller-manager-65484b555f-jrtff\" (UID: \"20becfa8-0f5d-490d-bf9f-2be53070a1b9\") " pod="openshift-jobset-operator/jobset-controller-manager-65484b555f-jrtff"
Oct 30 21:58:01.125872 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:58:01.125849    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/20becfa8-0f5d-490d-bf9f-2be53070a1b9-manager-config\") pod \"jobset-controller-manager-65484b555f-jrtff\" (UID: \"20becfa8-0f5d-490d-bf9f-2be53070a1b9\") " pod="openshift-jobset-operator/jobset-controller-manager-65484b555f-jrtff"
Oct 30 21:58:01.127559 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:58:01.127536    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/20becfa8-0f5d-490d-bf9f-2be53070a1b9-cert\") pod \"jobset-controller-manager-65484b555f-jrtff\" (UID: \"20becfa8-0f5d-490d-bf9f-2be53070a1b9\") " pod="openshift-jobset-operator/jobset-controller-manager-65484b555f-jrtff"
Oct 30 21:58:01.127727 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:58:01.127709    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20becfa8-0f5d-490d-bf9f-2be53070a1b9-metrics-certs\") pod \"jobset-controller-manager-65484b555f-jrtff\" (UID: \"20becfa8-0f5d-490d-bf9f-2be53070a1b9\") " pod="openshift-jobset-operator/jobset-controller-manager-65484b555f-jrtff"
Oct 30 21:58:01.183261 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:58:01.183194    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjt99\" (UniqueName: \"kubernetes.io/projected/20becfa8-0f5d-490d-bf9f-2be53070a1b9-kube-api-access-pjt99\") pod \"jobset-controller-manager-65484b555f-jrtff\" (UID: \"20becfa8-0f5d-490d-bf9f-2be53070a1b9\") " pod="openshift-jobset-operator/jobset-controller-manager-65484b555f-jrtff"
Oct 30 21:58:01.210101 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:58:01.210070    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-65484b555f-jrtff"
Oct 30 21:58:01.585333 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:58:01.585287    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-65484b555f-jrtff"]
Oct 30 21:58:01.587519 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:58:01.587487    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20becfa8_0f5d_490d_bf9f_2be53070a1b9.slice/crio-d3301595d3e21e5e04b3eedb64801a968562566a06854bd6d1305231f8d824cb WatchSource:0}: Error finding container d3301595d3e21e5e04b3eedb64801a968562566a06854bd6d1305231f8d824cb: Status 404 returned error can't find the container with id d3301595d3e21e5e04b3eedb64801a968562566a06854bd6d1305231f8d824cb
Oct 30 21:58:02.376778 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:58:02.376744    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-65484b555f-jrtff" event={"ID":"20becfa8-0f5d-490d-bf9f-2be53070a1b9","Type":"ContainerStarted","Data":"d3301595d3e21e5e04b3eedb64801a968562566a06854bd6d1305231f8d824cb"}
Oct 30 21:58:06.399734 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:58:06.399694    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-65484b555f-jrtff" event={"ID":"20becfa8-0f5d-490d-bf9f-2be53070a1b9","Type":"ContainerStarted","Data":"d3bdd72e42725fbc64e7a023e0373eeeb841bb026646a25c8250698648868ceb"}
Oct 30 21:58:06.400137 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:58:06.399769    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-65484b555f-jrtff"
Oct 30 21:58:06.513961 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:58:06.513899    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-65484b555f-jrtff" podStartSLOduration=2.247151876 podStartE2EDuration="6.513881488s" totalImagesPullingTime="4.266729612s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:06.512348587 +0000 UTC m=+464.116656139" watchObservedRunningTime="2026-10-30 21:58:06.513881488 +0000 UTC m=+464.118189045"
Oct 30 21:58:10.569999 ip-10-0-128-245 kubenswrapper[5678]: E1030 21:58:10.569965    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 21:58:17.408953 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:58:17.408923    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-65484b555f-jrtff"
Oct 30 21:58:40.573555 ip-10-0-128-245 kubenswrapper[5678]: E1030 21:58:40.573512    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 21:59:10.570572 ip-10-0-128-245 kubenswrapper[5678]: E1030 21:59:10.570478    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 21:59:25.682550 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:59:25.682511    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-8vwn4"]
Oct 30 21:59:25.685027 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:59:25.685010    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-8vwn4"
Oct 30 21:59:25.734833 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:59:25.734807    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Oct 30 21:59:25.781493 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:59:25.781463    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Oct 30 21:59:25.781640 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:59:25.781538    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-h2kw6\""
Oct 30 21:59:25.781640 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:59:25.781633    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Oct 30 21:59:25.783281 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:59:25.783258    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-8vwn4"]
Oct 30 21:59:25.824151 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:59:25.824123    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/057ad8c2-55df-4e8e-b599-e722ea8b9925-tls-certs\") pod \"model-serving-api-75bd89fdfc-8vwn4\" (UID: \"057ad8c2-55df-4e8e-b599-e722ea8b9925\") " pod="opendatahub/model-serving-api-75bd89fdfc-8vwn4"
Oct 30 21:59:25.824283 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:59:25.824266    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8n9ss\" (UniqueName: \"kubernetes.io/projected/057ad8c2-55df-4e8e-b599-e722ea8b9925-kube-api-access-8n9ss\") pod \"model-serving-api-75bd89fdfc-8vwn4\" (UID: \"057ad8c2-55df-4e8e-b599-e722ea8b9925\") " pod="opendatahub/model-serving-api-75bd89fdfc-8vwn4"
Oct 30 21:59:25.924829 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:59:25.924789    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8n9ss\" (UniqueName: \"kubernetes.io/projected/057ad8c2-55df-4e8e-b599-e722ea8b9925-kube-api-access-8n9ss\") pod \"model-serving-api-75bd89fdfc-8vwn4\" (UID: \"057ad8c2-55df-4e8e-b599-e722ea8b9925\") " pod="opendatahub/model-serving-api-75bd89fdfc-8vwn4"
Oct 30 21:59:25.924829 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:59:25.924831    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/057ad8c2-55df-4e8e-b599-e722ea8b9925-tls-certs\") pod \"model-serving-api-75bd89fdfc-8vwn4\" (UID: \"057ad8c2-55df-4e8e-b599-e722ea8b9925\") " pod="opendatahub/model-serving-api-75bd89fdfc-8vwn4"
Oct 30 21:59:25.927350 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:59:25.927324    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/057ad8c2-55df-4e8e-b599-e722ea8b9925-tls-certs\") pod \"model-serving-api-75bd89fdfc-8vwn4\" (UID: \"057ad8c2-55df-4e8e-b599-e722ea8b9925\") " pod="opendatahub/model-serving-api-75bd89fdfc-8vwn4"
Oct 30 21:59:25.983631 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:59:25.983565    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8n9ss\" (UniqueName: \"kubernetes.io/projected/057ad8c2-55df-4e8e-b599-e722ea8b9925-kube-api-access-8n9ss\") pod \"model-serving-api-75bd89fdfc-8vwn4\" (UID: \"057ad8c2-55df-4e8e-b599-e722ea8b9925\") " pod="opendatahub/model-serving-api-75bd89fdfc-8vwn4"
Oct 30 21:59:25.997371 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:59:25.997347    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-8vwn4"
Oct 30 21:59:26.370536 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:59:26.370447    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-8vwn4"]
Oct 30 21:59:26.373183 ip-10-0-128-245 kubenswrapper[5678]: W1030 21:59:26.373151    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod057ad8c2_55df_4e8e_b599_e722ea8b9925.slice/crio-66664018bba4e29c81979d1a324320fb923275e14cf6f5b93ef3c14956c31673 WatchSource:0}: Error finding container 66664018bba4e29c81979d1a324320fb923275e14cf6f5b93ef3c14956c31673: Status 404 returned error can't find the container with id 66664018bba4e29c81979d1a324320fb923275e14cf6f5b93ef3c14956c31673
Oct 30 21:59:26.755350 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:59:26.755276    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-8vwn4" event={"ID":"057ad8c2-55df-4e8e-b599-e722ea8b9925","Type":"ContainerStarted","Data":"66664018bba4e29c81979d1a324320fb923275e14cf6f5b93ef3c14956c31673"}
Oct 30 21:59:28.767373 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:59:28.767333    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-8vwn4" event={"ID":"057ad8c2-55df-4e8e-b599-e722ea8b9925","Type":"ContainerStarted","Data":"be18fda09dcf804fb12c5d02fda9939bce237101f4f35d6dd2af42713a5603bc"}
Oct 30 21:59:28.767819 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:59:28.767443    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-8vwn4"
Oct 30 21:59:28.892134 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:59:28.892014    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-8vwn4" podStartSLOduration=1.6460494049999999 podStartE2EDuration="3.891990402s" totalImagesPullingTime="2.245940997s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:59:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:59:28.891812752 +0000 UTC m=+546.496120360" watchObservedRunningTime="2026-10-30 21:59:28.891990402 +0000 UTC m=+546.496297952"
Oct 30 21:59:39.776808 ip-10-0-128-245 kubenswrapper[5678]: I1030 21:59:39.776729    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-8vwn4"
Oct 30 21:59:40.569550 ip-10-0-128-245 kubenswrapper[5678]: E1030 21:59:40.569513    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 22:00:10.570377 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:00:10.570340    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 22:00:40.571832 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:00:40.571796    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 22:01:10.571435 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:01:10.571345    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 22:01:40.570735 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:01:40.570699    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 22:01:55.344345 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:01:55.344311    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9ndd9"]
Oct 30 22:01:55.346753 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:01:55.346733    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9ndd9"
Oct 30 22:01:55.396443 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:01:55.396387    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Oct 30 22:01:55.442875 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:01:55.442844    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Oct 30 22:01:55.443074 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:01:55.442965    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Oct 30 22:01:55.443074 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:01:55.443039    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-f92sw\""
Oct 30 22:01:55.444496 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:01:55.444480    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9ndd9"]
Oct 30 22:01:55.453991 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:01:55.453973    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqb6w\" (UniqueName: \"kubernetes.io/projected/81bd6415-f42f-446b-a64e-956f794b3956-kube-api-access-tqb6w\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9ndd9\" (UID: \"81bd6415-f42f-446b-a64e-956f794b3956\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9ndd9"
Oct 30 22:01:55.454059 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:01:55.454004    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81bd6415-f42f-446b-a64e-956f794b3956-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9ndd9\" (UID: \"81bd6415-f42f-446b-a64e-956f794b3956\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9ndd9"
Oct 30 22:01:55.454059 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:01:55.454023    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81bd6415-f42f-446b-a64e-956f794b3956-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9ndd9\" (UID: \"81bd6415-f42f-446b-a64e-956f794b3956\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9ndd9"
Oct 30 22:01:55.454059 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:01:55.454038    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81bd6415-f42f-446b-a64e-956f794b3956-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9ndd9\" (UID: \"81bd6415-f42f-446b-a64e-956f794b3956\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9ndd9"
Oct 30 22:01:55.454215 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:01:55.454181    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81bd6415-f42f-446b-a64e-956f794b3956-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9ndd9\" (UID: \"81bd6415-f42f-446b-a64e-956f794b3956\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9ndd9"
Oct 30 22:01:55.454270 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:01:55.454246    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81bd6415-f42f-446b-a64e-956f794b3956-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9ndd9\" (UID: \"81bd6415-f42f-446b-a64e-956f794b3956\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9ndd9"
Oct 30 22:01:55.555544 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:01:55.555514    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81bd6415-f42f-446b-a64e-956f794b3956-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9ndd9\" (UID: \"81bd6415-f42f-446b-a64e-956f794b3956\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9ndd9"
Oct 30 22:01:55.555697 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:01:55.555575    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81bd6415-f42f-446b-a64e-956f794b3956-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9ndd9\" (UID: \"81bd6415-f42f-446b-a64e-956f794b3956\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9ndd9"
Oct 30 22:01:55.555697 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:01:55.555627    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqb6w\" (UniqueName: \"kubernetes.io/projected/81bd6415-f42f-446b-a64e-956f794b3956-kube-api-access-tqb6w\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9ndd9\" (UID: \"81bd6415-f42f-446b-a64e-956f794b3956\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9ndd9"
Oct 30 22:01:55.555813 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:01:55.555754    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81bd6415-f42f-446b-a64e-956f794b3956-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9ndd9\" (UID: \"81bd6415-f42f-446b-a64e-956f794b3956\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9ndd9"
Oct 30 22:01:55.555813 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:01:55.555790    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81bd6415-f42f-446b-a64e-956f794b3956-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9ndd9\" (UID: \"81bd6415-f42f-446b-a64e-956f794b3956\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9ndd9"
Oct 30 22:01:55.555813 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:01:55.555807    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81bd6415-f42f-446b-a64e-956f794b3956-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9ndd9\" (UID: \"81bd6415-f42f-446b-a64e-956f794b3956\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9ndd9"
Oct 30 22:01:55.556147 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:01:55.556121    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81bd6415-f42f-446b-a64e-956f794b3956-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9ndd9\" (UID: \"81bd6415-f42f-446b-a64e-956f794b3956\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9ndd9"
Oct 30 22:01:55.556147 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:01:55.556137    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81bd6415-f42f-446b-a64e-956f794b3956-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9ndd9\" (UID: \"81bd6415-f42f-446b-a64e-956f794b3956\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9ndd9"
Oct 30 22:01:55.556302 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:01:55.556174    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81bd6415-f42f-446b-a64e-956f794b3956-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9ndd9\" (UID: \"81bd6415-f42f-446b-a64e-956f794b3956\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9ndd9"
Oct 30 22:01:55.557771 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:01:55.557748    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81bd6415-f42f-446b-a64e-956f794b3956-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9ndd9\" (UID: \"81bd6415-f42f-446b-a64e-956f794b3956\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9ndd9"
Oct 30 22:01:55.558032 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:01:55.558015    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81bd6415-f42f-446b-a64e-956f794b3956-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9ndd9\" (UID: \"81bd6415-f42f-446b-a64e-956f794b3956\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9ndd9"
Oct 30 22:01:55.613878 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:01:55.613816    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqb6w\" (UniqueName: \"kubernetes.io/projected/81bd6415-f42f-446b-a64e-956f794b3956-kube-api-access-tqb6w\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9ndd9\" (UID: \"81bd6415-f42f-446b-a64e-956f794b3956\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9ndd9"
Oct 30 22:01:55.657132 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:01:55.657107    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9ndd9"
Oct 30 22:01:56.382732 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:01:56.382704    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9ndd9"]
Oct 30 22:01:56.385319 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:01:56.385290    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod81bd6415_f42f_446b_a64e_956f794b3956.slice/crio-266cf3cb83d3a3f7ffb51565fa764eb360f7bd9b4b7ab5ec3f4f9fc68ab4a51d WatchSource:0}: Error finding container 266cf3cb83d3a3f7ffb51565fa764eb360f7bd9b4b7ab5ec3f4f9fc68ab4a51d: Status 404 returned error can't find the container with id 266cf3cb83d3a3f7ffb51565fa764eb360f7bd9b4b7ab5ec3f4f9fc68ab4a51d
Oct 30 22:01:56.386683 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:01:56.386661    5678 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 22:01:56.424971 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:01:56.424936    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9ndd9" event={"ID":"81bd6415-f42f-446b-a64e-956f794b3956","Type":"ContainerStarted","Data":"266cf3cb83d3a3f7ffb51565fa764eb360f7bd9b4b7ab5ec3f4f9fc68ab4a51d"}
Oct 30 22:01:58.438455 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:01:58.438420    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9ndd9" event={"ID":"81bd6415-f42f-446b-a64e-956f794b3956","Type":"ContainerStarted","Data":"38ef61b6f1ce68f39e3112f49b66a92199212d5240a16c508ab0d9e148db3cd2"}
Oct 30 22:01:58.550555 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:01:58.550499    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9ndd9" podStartSLOduration=2.068276401 podStartE2EDuration="3.550483312s" totalImagesPullingTime="1.482206911s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:01:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:01:58.550284939 +0000 UTC m=+696.154592502" watchObservedRunningTime="2026-10-30 22:01:58.550483312 +0000 UTC m=+696.154790866"
Oct 30 22:01:58.658084 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:01:58.658054    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9ndd9"
Oct 30 22:02:03.173106 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:03.173070    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-plz4v"]
Oct 30 22:02:03.175629 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:03.175612    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-plz4v"
Oct 30 22:02:03.220319 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:03.220284    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9z95\" (UniqueName: \"kubernetes.io/projected/c32955de-27bc-4ed0-83ab-7cf9f4a8b6cf-kube-api-access-r9z95\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-plz4v\" (UID: \"c32955de-27bc-4ed0-83ab-7cf9f4a8b6cf\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-plz4v"
Oct 30 22:02:03.220501 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:03.220330    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c32955de-27bc-4ed0-83ab-7cf9f4a8b6cf-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-plz4v\" (UID: \"c32955de-27bc-4ed0-83ab-7cf9f4a8b6cf\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-plz4v"
Oct 30 22:02:03.220501 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:03.220353    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c32955de-27bc-4ed0-83ab-7cf9f4a8b6cf-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-plz4v\" (UID: \"c32955de-27bc-4ed0-83ab-7cf9f4a8b6cf\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-plz4v"
Oct 30 22:02:03.220626 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:03.220528    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c32955de-27bc-4ed0-83ab-7cf9f4a8b6cf-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-plz4v\" (UID: \"c32955de-27bc-4ed0-83ab-7cf9f4a8b6cf\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-plz4v"
Oct 30 22:02:03.220626 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:03.220573    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c32955de-27bc-4ed0-83ab-7cf9f4a8b6cf-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-plz4v\" (UID: \"c32955de-27bc-4ed0-83ab-7cf9f4a8b6cf\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-plz4v"
Oct 30 22:02:03.220731 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:03.220645    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c32955de-27bc-4ed0-83ab-7cf9f4a8b6cf-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-plz4v\" (UID: \"c32955de-27bc-4ed0-83ab-7cf9f4a8b6cf\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-plz4v"
Oct 30 22:02:03.272133 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:03.272106    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Oct 30 22:02:03.273594 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:03.273572    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-plz4v"]
Oct 30 22:02:03.321461 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:03.321423    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c32955de-27bc-4ed0-83ab-7cf9f4a8b6cf-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-plz4v\" (UID: \"c32955de-27bc-4ed0-83ab-7cf9f4a8b6cf\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-plz4v"
Oct 30 22:02:03.321626 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:03.321499    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9z95\" (UniqueName: \"kubernetes.io/projected/c32955de-27bc-4ed0-83ab-7cf9f4a8b6cf-kube-api-access-r9z95\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-plz4v\" (UID: \"c32955de-27bc-4ed0-83ab-7cf9f4a8b6cf\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-plz4v"
Oct 30 22:02:03.321626 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:03.321522    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c32955de-27bc-4ed0-83ab-7cf9f4a8b6cf-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-plz4v\" (UID: \"c32955de-27bc-4ed0-83ab-7cf9f4a8b6cf\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-plz4v"
Oct 30 22:02:03.321626 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:03.321539    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c32955de-27bc-4ed0-83ab-7cf9f4a8b6cf-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-plz4v\" (UID: \"c32955de-27bc-4ed0-83ab-7cf9f4a8b6cf\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-plz4v"
Oct 30 22:02:03.321626 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:03.321576    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c32955de-27bc-4ed0-83ab-7cf9f4a8b6cf-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-plz4v\" (UID: \"c32955de-27bc-4ed0-83ab-7cf9f4a8b6cf\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-plz4v"
Oct 30 22:02:03.321626 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:03.321604    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c32955de-27bc-4ed0-83ab-7cf9f4a8b6cf-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-plz4v\" (UID: \"c32955de-27bc-4ed0-83ab-7cf9f4a8b6cf\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-plz4v"
Oct 30 22:02:03.321885 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:03.321810    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c32955de-27bc-4ed0-83ab-7cf9f4a8b6cf-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-plz4v\" (UID: \"c32955de-27bc-4ed0-83ab-7cf9f4a8b6cf\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-plz4v"
Oct 30 22:02:03.321958 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:03.321902    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c32955de-27bc-4ed0-83ab-7cf9f4a8b6cf-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-plz4v\" (UID: \"c32955de-27bc-4ed0-83ab-7cf9f4a8b6cf\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-plz4v"
Oct 30 22:02:03.322014 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:03.321979    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c32955de-27bc-4ed0-83ab-7cf9f4a8b6cf-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-plz4v\" (UID: \"c32955de-27bc-4ed0-83ab-7cf9f4a8b6cf\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-plz4v"
Oct 30 22:02:03.323793 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:03.323769    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c32955de-27bc-4ed0-83ab-7cf9f4a8b6cf-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-plz4v\" (UID: \"c32955de-27bc-4ed0-83ab-7cf9f4a8b6cf\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-plz4v"
Oct 30 22:02:03.323989 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:03.323969    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c32955de-27bc-4ed0-83ab-7cf9f4a8b6cf-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-plz4v\" (UID: \"c32955de-27bc-4ed0-83ab-7cf9f4a8b6cf\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-plz4v"
Oct 30 22:02:03.380302 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:03.380268    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9z95\" (UniqueName: \"kubernetes.io/projected/c32955de-27bc-4ed0-83ab-7cf9f4a8b6cf-kube-api-access-r9z95\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-plz4v\" (UID: \"c32955de-27bc-4ed0-83ab-7cf9f4a8b6cf\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-plz4v"
Oct 30 22:02:03.488007 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:03.487920    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-plz4v"
Oct 30 22:02:03.865477 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:03.865449    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-plz4v"]
Oct 30 22:02:03.868577 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:02:03.868552    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc32955de_27bc_4ed0_83ab_7cf9f4a8b6cf.slice/crio-644c51478c3a81d8360fcf4f28f2d7cbe50b31589ebfd7c1ebc2c16182fadf58 WatchSource:0}: Error finding container 644c51478c3a81d8360fcf4f28f2d7cbe50b31589ebfd7c1ebc2c16182fadf58: Status 404 returned error can't find the container with id 644c51478c3a81d8360fcf4f28f2d7cbe50b31589ebfd7c1ebc2c16182fadf58
Oct 30 22:02:04.466689 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:04.466662    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-plz4v" event={"ID":"c32955de-27bc-4ed0-83ab-7cf9f4a8b6cf","Type":"ContainerStarted","Data":"644c51478c3a81d8360fcf4f28f2d7cbe50b31589ebfd7c1ebc2c16182fadf58"}
Oct 30 22:02:05.472878 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:05.472839    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-plz4v" event={"ID":"c32955de-27bc-4ed0-83ab-7cf9f4a8b6cf","Type":"ContainerStarted","Data":"e6b916449b96661f8308c5457bdf026d61299d7ac4e5a872e05e84223d8fd7fa"}
Oct 30 22:02:05.488133 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:05.488113    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-plz4v"
Oct 30 22:02:05.587898 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:05.587850    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-plz4v" podStartSLOduration=2.012844799 podStartE2EDuration="2.587838009s" totalImagesPullingTime="574.99321ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:02:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:02:05.587313029 +0000 UTC m=+703.191620593" watchObservedRunningTime="2026-10-30 22:02:05.587838009 +0000 UTC m=+703.192145562"
Oct 30 22:02:05.657322 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:05.657290    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9ndd9"
Oct 30 22:02:05.669639 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:05.669614    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9ndd9"
Oct 30 22:02:06.488422 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:06.488383    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9ndd9"
Oct 30 22:02:10.570848 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:02:10.570813    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 22:02:10.812178 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:10.812145    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-klr4q"]
Oct 30 22:02:10.814836 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:10.814816    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-klr4q"
Oct 30 22:02:10.884906 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:10.884878    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b10aa0a-17b4-427e-a9d8-b9202ba1ac0a-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-klr4q\" (UID: \"0b10aa0a-17b4-427e-a9d8-b9202ba1ac0a\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-klr4q"
Oct 30 22:02:10.885075 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:10.884912    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b10aa0a-17b4-427e-a9d8-b9202ba1ac0a-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-klr4q\" (UID: \"0b10aa0a-17b4-427e-a9d8-b9202ba1ac0a\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-klr4q"
Oct 30 22:02:10.885075 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:10.884935    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8h7jk\" (UniqueName: \"kubernetes.io/projected/0b10aa0a-17b4-427e-a9d8-b9202ba1ac0a-kube-api-access-8h7jk\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-klr4q\" (UID: \"0b10aa0a-17b4-427e-a9d8-b9202ba1ac0a\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-klr4q"
Oct 30 22:02:10.885194 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:10.885098    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b10aa0a-17b4-427e-a9d8-b9202ba1ac0a-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-klr4q\" (UID: \"0b10aa0a-17b4-427e-a9d8-b9202ba1ac0a\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-klr4q"
Oct 30 22:02:10.885194 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:10.885185    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b10aa0a-17b4-427e-a9d8-b9202ba1ac0a-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-klr4q\" (UID: \"0b10aa0a-17b4-427e-a9d8-b9202ba1ac0a\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-klr4q"
Oct 30 22:02:10.885277 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:10.885244    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b10aa0a-17b4-427e-a9d8-b9202ba1ac0a-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-klr4q\" (UID: \"0b10aa0a-17b4-427e-a9d8-b9202ba1ac0a\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-klr4q"
Oct 30 22:02:10.911158 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:10.911131    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Oct 30 22:02:10.912685 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:10.912663    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-klr4q"]
Oct 30 22:02:10.986263 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:10.986228    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b10aa0a-17b4-427e-a9d8-b9202ba1ac0a-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-klr4q\" (UID: \"0b10aa0a-17b4-427e-a9d8-b9202ba1ac0a\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-klr4q"
Oct 30 22:02:10.986263 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:10.986265    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b10aa0a-17b4-427e-a9d8-b9202ba1ac0a-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-klr4q\" (UID: \"0b10aa0a-17b4-427e-a9d8-b9202ba1ac0a\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-klr4q"
Oct 30 22:02:10.986517 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:10.986305    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b10aa0a-17b4-427e-a9d8-b9202ba1ac0a-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-klr4q\" (UID: \"0b10aa0a-17b4-427e-a9d8-b9202ba1ac0a\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-klr4q"
Oct 30 22:02:10.986517 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:10.986334    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8h7jk\" (UniqueName: \"kubernetes.io/projected/0b10aa0a-17b4-427e-a9d8-b9202ba1ac0a-kube-api-access-8h7jk\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-klr4q\" (UID: \"0b10aa0a-17b4-427e-a9d8-b9202ba1ac0a\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-klr4q"
Oct 30 22:02:10.986517 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:10.986378    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b10aa0a-17b4-427e-a9d8-b9202ba1ac0a-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-klr4q\" (UID: \"0b10aa0a-17b4-427e-a9d8-b9202ba1ac0a\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-klr4q"
Oct 30 22:02:10.986517 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:10.986450    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b10aa0a-17b4-427e-a9d8-b9202ba1ac0a-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-klr4q\" (UID: \"0b10aa0a-17b4-427e-a9d8-b9202ba1ac0a\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-klr4q"
Oct 30 22:02:10.986731 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:10.986711    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b10aa0a-17b4-427e-a9d8-b9202ba1ac0a-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-klr4q\" (UID: \"0b10aa0a-17b4-427e-a9d8-b9202ba1ac0a\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-klr4q"
Oct 30 22:02:10.986774 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:10.986757    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b10aa0a-17b4-427e-a9d8-b9202ba1ac0a-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-klr4q\" (UID: \"0b10aa0a-17b4-427e-a9d8-b9202ba1ac0a\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-klr4q"
Oct 30 22:02:10.986836 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:10.986774    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b10aa0a-17b4-427e-a9d8-b9202ba1ac0a-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-klr4q\" (UID: \"0b10aa0a-17b4-427e-a9d8-b9202ba1ac0a\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-klr4q"
Oct 30 22:02:10.988693 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:10.988669    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b10aa0a-17b4-427e-a9d8-b9202ba1ac0a-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-klr4q\" (UID: \"0b10aa0a-17b4-427e-a9d8-b9202ba1ac0a\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-klr4q"
Oct 30 22:02:10.988836 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:10.988820    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b10aa0a-17b4-427e-a9d8-b9202ba1ac0a-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-klr4q\" (UID: \"0b10aa0a-17b4-427e-a9d8-b9202ba1ac0a\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-klr4q"
Oct 30 22:02:11.045393 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:11.045353    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8h7jk\" (UniqueName: \"kubernetes.io/projected/0b10aa0a-17b4-427e-a9d8-b9202ba1ac0a-kube-api-access-8h7jk\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-klr4q\" (UID: \"0b10aa0a-17b4-427e-a9d8-b9202ba1ac0a\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-klr4q"
Oct 30 22:02:11.127210 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:11.127173    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-klr4q"
Oct 30 22:02:11.503225 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:11.503202    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-klr4q"]
Oct 30 22:02:11.505651 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:02:11.505623    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b10aa0a_17b4_427e_a9d8_b9202ba1ac0a.slice/crio-8e0d5bb2d2732ec105cf5376c38c6f405ce2c50f25ef84267c681179050c9711 WatchSource:0}: Error finding container 8e0d5bb2d2732ec105cf5376c38c6f405ce2c50f25ef84267c681179050c9711: Status 404 returned error can't find the container with id 8e0d5bb2d2732ec105cf5376c38c6f405ce2c50f25ef84267c681179050c9711
Oct 30 22:02:12.507629 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:12.507591    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-klr4q" event={"ID":"0b10aa0a-17b4-427e-a9d8-b9202ba1ac0a","Type":"ContainerStarted","Data":"e8117fab1033f5992d76d731f6c57e38127d16dfa1837916f7716d022877b915"}
Oct 30 22:02:12.508137 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:12.507656    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-klr4q" event={"ID":"0b10aa0a-17b4-427e-a9d8-b9202ba1ac0a","Type":"ContainerStarted","Data":"8e0d5bb2d2732ec105cf5376c38c6f405ce2c50f25ef84267c681179050c9711"}
Oct 30 22:02:12.621152 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:12.621098    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-klr4q" podStartSLOduration=2.052774912 podStartE2EDuration="2.621085727s" totalImagesPullingTime="568.310815ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:02:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:02:12.619846521 +0000 UTC m=+710.224154076" watchObservedRunningTime="2026-10-30 22:02:12.621085727 +0000 UTC m=+710.225393324"
Oct 30 22:02:13.127619 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:13.127589    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-klr4q"
Oct 30 22:02:13.488833 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:13.488744    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-plz4v"
Oct 30 22:02:13.501201 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:13.501180    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-plz4v"
Oct 30 22:02:13.523627 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:13.523602    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-plz4v"
Oct 30 22:02:21.127437 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:21.127391    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-klr4q"
Oct 30 22:02:21.139843 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:21.139820    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-klr4q"
Oct 30 22:02:21.316253 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:21.316226    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-jsqpx"]
Oct 30 22:02:21.318628 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:21.318612    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8mdvr"]
Oct 30 22:02:21.318769 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:21.318747    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-jsqpx"
Oct 30 22:02:21.326912 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:21.326878    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8mdvr"
Oct 30 22:02:21.368908 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:21.368883    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 30 22:02:21.420620 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:21.420552    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-v8cwr\""
Oct 30 22:02:21.420771 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:21.420638    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8mdvr"]
Oct 30 22:02:21.425496 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:21.425469    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-jsqpx"]
Oct 30 22:02:21.479673 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:21.479644    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfbe138a-ae9f-45e4-9b30-1bf33a80929c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-jsqpx\" (UID: \"bfbe138a-ae9f-45e4-9b30-1bf33a80929c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-jsqpx"
Oct 30 22:02:21.479673 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:21.479678    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfbe138a-ae9f-45e4-9b30-1bf33a80929c-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-jsqpx\" (UID: \"bfbe138a-ae9f-45e4-9b30-1bf33a80929c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-jsqpx"
Oct 30 22:02:21.479858 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:21.479696    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4729d87-68c3-4c11-9a1d-c235bc370104-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-8mdvr\" (UID: \"c4729d87-68c3-4c11-9a1d-c235bc370104\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8mdvr"
Oct 30 22:02:21.479858 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:21.479716    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4729d87-68c3-4c11-9a1d-c235bc370104-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-8mdvr\" (UID: \"c4729d87-68c3-4c11-9a1d-c235bc370104\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8mdvr"
Oct 30 22:02:21.479858 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:21.479768    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfbe138a-ae9f-45e4-9b30-1bf33a80929c-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-jsqpx\" (UID: \"bfbe138a-ae9f-45e4-9b30-1bf33a80929c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-jsqpx"
Oct 30 22:02:21.479957 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:21.479878    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4729d87-68c3-4c11-9a1d-c235bc370104-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-8mdvr\" (UID: \"c4729d87-68c3-4c11-9a1d-c235bc370104\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8mdvr"
Oct 30 22:02:21.479957 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:21.479910    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmnts\" (UniqueName: \"kubernetes.io/projected/c4729d87-68c3-4c11-9a1d-c235bc370104-kube-api-access-mmnts\") pod \"maas-default-gateway-openshift-default-85544d4cd7-8mdvr\" (UID: \"c4729d87-68c3-4c11-9a1d-c235bc370104\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8mdvr"
Oct 30 22:02:21.479957 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:21.479945    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4729d87-68c3-4c11-9a1d-c235bc370104-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-8mdvr\" (UID: \"c4729d87-68c3-4c11-9a1d-c235bc370104\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8mdvr"
Oct 30 22:02:21.480060 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:21.479988    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfbe138a-ae9f-45e4-9b30-1bf33a80929c-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-jsqpx\" (UID: \"bfbe138a-ae9f-45e4-9b30-1bf33a80929c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-jsqpx"
Oct 30 22:02:21.480060 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:21.480025    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfbe138a-ae9f-45e4-9b30-1bf33a80929c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-jsqpx\" (UID: \"bfbe138a-ae9f-45e4-9b30-1bf33a80929c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-jsqpx"
Oct 30 22:02:21.480122 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:21.480074    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4729d87-68c3-4c11-9a1d-c235bc370104-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-8mdvr\" (UID: \"c4729d87-68c3-4c11-9a1d-c235bc370104\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8mdvr"
Oct 30 22:02:21.480122 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:21.480113    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4729d87-68c3-4c11-9a1d-c235bc370104-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-8mdvr\" (UID: \"c4729d87-68c3-4c11-9a1d-c235bc370104\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8mdvr"
Oct 30 22:02:21.480182 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:21.480129    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfbe138a-ae9f-45e4-9b30-1bf33a80929c-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-jsqpx\" (UID: \"bfbe138a-ae9f-45e4-9b30-1bf33a80929c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-jsqpx"
Oct 30 22:02:21.480182 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:21.480146    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfbe138a-ae9f-45e4-9b30-1bf33a80929c-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-jsqpx\" (UID: \"bfbe138a-ae9f-45e4-9b30-1bf33a80929c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-jsqpx"
Oct 30 22:02:21.480182 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:21.480160    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4729d87-68c3-4c11-9a1d-c235bc370104-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-8mdvr\" (UID: \"c4729d87-68c3-4c11-9a1d-c235bc370104\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8mdvr"
Oct 30 22:02:21.480270 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:21.480209    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfbe138a-ae9f-45e4-9b30-1bf33a80929c-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-jsqpx\" (UID: \"bfbe138a-ae9f-45e4-9b30-1bf33a80929c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-jsqpx"
Oct 30 22:02:21.480270 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:21.480234    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8q4gn\" (UniqueName: \"kubernetes.io/projected/bfbe138a-ae9f-45e4-9b30-1bf33a80929c-kube-api-access-8q4gn\") pod \"maas-default-gateway-openshift-default-85544d4cd7-jsqpx\" (UID: \"bfbe138a-ae9f-45e4-9b30-1bf33a80929c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-jsqpx"
Oct 30 22:02:21.480333 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:21.480282    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4729d87-68c3-4c11-9a1d-c235bc370104-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-8mdvr\" (UID: \"c4729d87-68c3-4c11-9a1d-c235bc370104\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8mdvr"
Oct 30 22:02:21.565173 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:21.565143    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-klr4q"
Oct 30 22:02:21.581046 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:21.581020    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4729d87-68c3-4c11-9a1d-c235bc370104-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-8mdvr\" (UID: \"c4729d87-68c3-4c11-9a1d-c235bc370104\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8mdvr"
Oct 30 22:02:21.581210 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:21.581051    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfbe138a-ae9f-45e4-9b30-1bf33a80929c-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-jsqpx\" (UID: \"bfbe138a-ae9f-45e4-9b30-1bf33a80929c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-jsqpx"
Oct 30 22:02:21.581210 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:21.581067    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfbe138a-ae9f-45e4-9b30-1bf33a80929c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-jsqpx\" (UID: \"bfbe138a-ae9f-45e4-9b30-1bf33a80929c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-jsqpx"
Oct 30 22:02:21.581210 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:21.581085    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4729d87-68c3-4c11-9a1d-c235bc370104-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-8mdvr\" (UID: \"c4729d87-68c3-4c11-9a1d-c235bc370104\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8mdvr"
Oct 30 22:02:21.581210 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:21.581101    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4729d87-68c3-4c11-9a1d-c235bc370104-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-8mdvr\" (UID: \"c4729d87-68c3-4c11-9a1d-c235bc370104\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8mdvr"
Oct 30 22:02:21.581210 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:21.581125    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfbe138a-ae9f-45e4-9b30-1bf33a80929c-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-jsqpx\" (UID: \"bfbe138a-ae9f-45e4-9b30-1bf33a80929c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-jsqpx"
Oct 30 22:02:21.581210 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:21.581151    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfbe138a-ae9f-45e4-9b30-1bf33a80929c-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-jsqpx\" (UID: \"bfbe138a-ae9f-45e4-9b30-1bf33a80929c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-jsqpx"
Oct 30 22:02:21.581210 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:21.581173    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4729d87-68c3-4c11-9a1d-c235bc370104-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-8mdvr\" (UID: \"c4729d87-68c3-4c11-9a1d-c235bc370104\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8mdvr"
Oct 30 22:02:21.581643 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:21.581221    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfbe138a-ae9f-45e4-9b30-1bf33a80929c-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-jsqpx\" (UID: \"bfbe138a-ae9f-45e4-9b30-1bf33a80929c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-jsqpx"
Oct 30 22:02:21.581643 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:21.581245    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8q4gn\" (UniqueName: \"kubernetes.io/projected/bfbe138a-ae9f-45e4-9b30-1bf33a80929c-kube-api-access-8q4gn\") pod \"maas-default-gateway-openshift-default-85544d4cd7-jsqpx\" (UID: \"bfbe138a-ae9f-45e4-9b30-1bf33a80929c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-jsqpx"
Oct 30 22:02:21.581643 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:21.581270    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4729d87-68c3-4c11-9a1d-c235bc370104-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-8mdvr\" (UID: \"c4729d87-68c3-4c11-9a1d-c235bc370104\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8mdvr"
Oct 30 22:02:21.581643 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:21.581296    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfbe138a-ae9f-45e4-9b30-1bf33a80929c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-jsqpx\" (UID: \"bfbe138a-ae9f-45e4-9b30-1bf33a80929c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-jsqpx"
Oct 30 22:02:21.581643 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:21.581323    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfbe138a-ae9f-45e4-9b30-1bf33a80929c-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-jsqpx\" (UID: \"bfbe138a-ae9f-45e4-9b30-1bf33a80929c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-jsqpx"
Oct 30 22:02:21.581643 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:21.581353    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4729d87-68c3-4c11-9a1d-c235bc370104-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-8mdvr\" (UID: \"c4729d87-68c3-4c11-9a1d-c235bc370104\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8mdvr"
Oct 30 22:02:21.581643 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:21.581382    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4729d87-68c3-4c11-9a1d-c235bc370104-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-8mdvr\" (UID: \"c4729d87-68c3-4c11-9a1d-c235bc370104\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8mdvr"
Oct 30 22:02:21.581643 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:21.581423    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfbe138a-ae9f-45e4-9b30-1bf33a80929c-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-jsqpx\" (UID: \"bfbe138a-ae9f-45e4-9b30-1bf33a80929c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-jsqpx"
Oct 30 22:02:21.581643 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:21.581490    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4729d87-68c3-4c11-9a1d-c235bc370104-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-8mdvr\" (UID: \"c4729d87-68c3-4c11-9a1d-c235bc370104\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8mdvr"
Oct 30 22:02:21.581643 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:21.581531    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfbe138a-ae9f-45e4-9b30-1bf33a80929c-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-jsqpx\" (UID: \"bfbe138a-ae9f-45e4-9b30-1bf33a80929c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-jsqpx"
Oct 30 22:02:21.581643 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:21.581546    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmnts\" (UniqueName: \"kubernetes.io/projected/c4729d87-68c3-4c11-9a1d-c235bc370104-kube-api-access-mmnts\") pod \"maas-default-gateway-openshift-default-85544d4cd7-8mdvr\" (UID: \"c4729d87-68c3-4c11-9a1d-c235bc370104\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8mdvr"
Oct 30 22:02:21.581643 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:21.581568    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4729d87-68c3-4c11-9a1d-c235bc370104-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-8mdvr\" (UID: \"c4729d87-68c3-4c11-9a1d-c235bc370104\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8mdvr"
Oct 30 22:02:21.581643 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:21.581603    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfbe138a-ae9f-45e4-9b30-1bf33a80929c-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-jsqpx\" (UID: \"bfbe138a-ae9f-45e4-9b30-1bf33a80929c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-jsqpx"
Oct 30 22:02:21.581643 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:21.581604    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4729d87-68c3-4c11-9a1d-c235bc370104-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-8mdvr\" (UID: \"c4729d87-68c3-4c11-9a1d-c235bc370104\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8mdvr"
Oct 30 22:02:21.582462 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:21.581800    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfbe138a-ae9f-45e4-9b30-1bf33a80929c-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-jsqpx\" (UID: \"bfbe138a-ae9f-45e4-9b30-1bf33a80929c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-jsqpx"
Oct 30 22:02:21.582462 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:21.581850    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4729d87-68c3-4c11-9a1d-c235bc370104-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-8mdvr\" (UID: \"c4729d87-68c3-4c11-9a1d-c235bc370104\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8mdvr"
Oct 30 22:02:21.582462 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:21.582185    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4729d87-68c3-4c11-9a1d-c235bc370104-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-8mdvr\" (UID: \"c4729d87-68c3-4c11-9a1d-c235bc370104\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8mdvr"
Oct 30 22:02:21.582462 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:21.582216    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfbe138a-ae9f-45e4-9b30-1bf33a80929c-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-jsqpx\" (UID: \"bfbe138a-ae9f-45e4-9b30-1bf33a80929c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-jsqpx"
Oct 30 22:02:21.582697 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:21.582470    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4729d87-68c3-4c11-9a1d-c235bc370104-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-8mdvr\" (UID: \"c4729d87-68c3-4c11-9a1d-c235bc370104\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8mdvr"
Oct 30 22:02:21.582796 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:21.582775    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfbe138a-ae9f-45e4-9b30-1bf33a80929c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-jsqpx\" (UID: \"bfbe138a-ae9f-45e4-9b30-1bf33a80929c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-jsqpx"
Oct 30 22:02:21.583871 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:21.583844    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4729d87-68c3-4c11-9a1d-c235bc370104-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-8mdvr\" (UID: \"c4729d87-68c3-4c11-9a1d-c235bc370104\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8mdvr"
Oct 30 22:02:21.583965 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:21.583878    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfbe138a-ae9f-45e4-9b30-1bf33a80929c-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-jsqpx\" (UID: \"bfbe138a-ae9f-45e4-9b30-1bf33a80929c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-jsqpx"
Oct 30 22:02:21.583965 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:21.583883    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4729d87-68c3-4c11-9a1d-c235bc370104-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-8mdvr\" (UID: \"c4729d87-68c3-4c11-9a1d-c235bc370104\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8mdvr"
Oct 30 22:02:21.584142 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:21.584127    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfbe138a-ae9f-45e4-9b30-1bf33a80929c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-jsqpx\" (UID: \"bfbe138a-ae9f-45e4-9b30-1bf33a80929c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-jsqpx"
Oct 30 22:02:21.639621 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:21.639590    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4729d87-68c3-4c11-9a1d-c235bc370104-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-8mdvr\" (UID: \"c4729d87-68c3-4c11-9a1d-c235bc370104\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8mdvr"
Oct 30 22:02:21.639715 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:21.639663    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfbe138a-ae9f-45e4-9b30-1bf33a80929c-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-jsqpx\" (UID: \"bfbe138a-ae9f-45e4-9b30-1bf33a80929c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-jsqpx"
Oct 30 22:02:21.639787 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:21.639772    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmnts\" (UniqueName: \"kubernetes.io/projected/c4729d87-68c3-4c11-9a1d-c235bc370104-kube-api-access-mmnts\") pod \"maas-default-gateway-openshift-default-85544d4cd7-8mdvr\" (UID: \"c4729d87-68c3-4c11-9a1d-c235bc370104\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8mdvr"
Oct 30 22:02:21.644688 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:21.644667    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8q4gn\" (UniqueName: \"kubernetes.io/projected/bfbe138a-ae9f-45e4-9b30-1bf33a80929c-kube-api-access-8q4gn\") pod \"maas-default-gateway-openshift-default-85544d4cd7-jsqpx\" (UID: \"bfbe138a-ae9f-45e4-9b30-1bf33a80929c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-jsqpx"
Oct 30 22:02:21.933674 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:21.933644    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-jsqpx"
Oct 30 22:02:21.939773 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:21.939750    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8mdvr"
Oct 30 22:02:22.365340 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:22.365317    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-jsqpx"]
Oct 30 22:02:22.367497 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:02:22.367457    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfbe138a_ae9f_45e4_9b30_1bf33a80929c.slice/crio-d54d4a22287506a666214b9dad819e79e31e3cfd6ed010430604262cc8d01854 WatchSource:0}: Error finding container d54d4a22287506a666214b9dad819e79e31e3cfd6ed010430604262cc8d01854: Status 404 returned error can't find the container with id d54d4a22287506a666214b9dad819e79e31e3cfd6ed010430604262cc8d01854
Oct 30 22:02:22.367831 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:22.367813    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8mdvr"]
Oct 30 22:02:22.404164 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:02:22.404128    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4729d87_68c3_4c11_9a1d_c235bc370104.slice/crio-afeca897c53940472a270df97daed8ed27cb80b7fee955731af861ea928c8955 WatchSource:0}: Error finding container afeca897c53940472a270df97daed8ed27cb80b7fee955731af861ea928c8955: Status 404 returned error can't find the container with id afeca897c53940472a270df97daed8ed27cb80b7fee955731af861ea928c8955
Oct 30 22:02:22.555191 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:22.555155    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8mdvr" event={"ID":"c4729d87-68c3-4c11-9a1d-c235bc370104","Type":"ContainerStarted","Data":"afeca897c53940472a270df97daed8ed27cb80b7fee955731af861ea928c8955"}
Oct 30 22:02:22.556416 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:22.556377    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-jsqpx" event={"ID":"bfbe138a-ae9f-45e4-9b30-1bf33a80929c","Type":"ContainerStarted","Data":"d54d4a22287506a666214b9dad819e79e31e3cfd6ed010430604262cc8d01854"}
Oct 30 22:02:26.966332 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:26.966302    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:02:26.966640 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:26.966367    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:02:26.966640 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:26.966416    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:02:26.975007 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:26.974972    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:02:26.975066 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:26.975056    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:02:26.975112 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:26.975095    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:02:27.584593 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:27.584559    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8mdvr" event={"ID":"c4729d87-68c3-4c11-9a1d-c235bc370104","Type":"ContainerStarted","Data":"6519165a5563672f1a81447ff1acdf2a7c5425518f74956226cdf4c4af390d16"}
Oct 30 22:02:27.586663 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:27.586631    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-jsqpx" event={"ID":"bfbe138a-ae9f-45e4-9b30-1bf33a80929c","Type":"ContainerStarted","Data":"4e9990f35796c3fe9b7fd654106c892e4c66690f5b7599b08bf7fd91b07761aa"}
Oct 30 22:02:27.808070 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:27.808017    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8mdvr" podStartSLOduration=2.23976162 podStartE2EDuration="6.808001958s" totalImagesPullingTime="4.568240338s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:02:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:02:27.75988566 +0000 UTC m=+725.364193215" watchObservedRunningTime="2026-10-30 22:02:27.808001958 +0000 UTC m=+725.412309511"
Oct 30 22:02:27.877965 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:27.877907    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-jsqpx" podStartSLOduration=2.281939171 podStartE2EDuration="6.877891534s" totalImagesPullingTime="4.595952363s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:02:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:02:27.875924333 +0000 UTC m=+725.480231886" watchObservedRunningTime="2026-10-30 22:02:27.877891534 +0000 UTC m=+725.482199086"
Oct 30 22:02:27.937559 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:27.937516    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-jsqpx"
Oct 30 22:02:27.945542 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:27.945504    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8mdvr"
Oct 30 22:02:28.949544 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:28.949508    5678 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-8mdvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Oct 30 22:02:28.950293 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:28.950076    5678 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-jsqpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Oct 30 22:02:28.950293 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:28.950236    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-jsqpx" podUID="bfbe138a-ae9f-45e4-9b30-1bf33a80929c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Oct 30 22:02:28.950787 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:28.950165    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8mdvr" podUID="c4729d87-68c3-4c11-9a1d-c235bc370104" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Oct 30 22:02:29.942427 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:29.939919    5678 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-jsqpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Oct 30 22:02:29.942427 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:29.939979    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-jsqpx" podUID="bfbe138a-ae9f-45e4-9b30-1bf33a80929c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Oct 30 22:02:29.942427 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:29.940599    5678 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-8mdvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Oct 30 22:02:29.942427 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:29.940637    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8mdvr" podUID="c4729d87-68c3-4c11-9a1d-c235bc370104" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Oct 30 22:02:30.934838 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:30.934799    5678 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-jsqpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Oct 30 22:02:30.935460 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:30.935424    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-jsqpx" podUID="bfbe138a-ae9f-45e4-9b30-1bf33a80929c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Oct 30 22:02:30.943588 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:30.943490    5678 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-8mdvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Oct 30 22:02:30.943588 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:30.943535    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8mdvr" podUID="c4729d87-68c3-4c11-9a1d-c235bc370104" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Oct 30 22:02:31.937865 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:31.937824    5678 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-jsqpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Oct 30 22:02:31.938633 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:31.938598    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-jsqpx" podUID="bfbe138a-ae9f-45e4-9b30-1bf33a80929c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Oct 30 22:02:31.942756 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:31.942731    5678 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-8mdvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Oct 30 22:02:31.942943 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:31.942918    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8mdvr" podUID="c4729d87-68c3-4c11-9a1d-c235bc370104" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Oct 30 22:02:32.937931 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:32.937894    5678 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-jsqpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Oct 30 22:02:32.938594 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:32.938544    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-jsqpx" podUID="bfbe138a-ae9f-45e4-9b30-1bf33a80929c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Oct 30 22:02:32.942738 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:32.942711    5678 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-8mdvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Oct 30 22:02:32.942949 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:32.942924    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8mdvr" podUID="c4729d87-68c3-4c11-9a1d-c235bc370104" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Oct 30 22:02:33.934759 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:33.934723    5678 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-jsqpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Oct 30 22:02:33.934967 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:33.934774    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-jsqpx" podUID="bfbe138a-ae9f-45e4-9b30-1bf33a80929c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Oct 30 22:02:33.940082 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:33.940054    5678 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-8mdvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Oct 30 22:02:33.940529 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:33.940109    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8mdvr" podUID="c4729d87-68c3-4c11-9a1d-c235bc370104" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Oct 30 22:02:35.297271 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:35.297236    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8mdvr"
Oct 30 22:02:35.297774 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:35.297691    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8mdvr"
Oct 30 22:02:35.298583 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:35.298553    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8mdvr"
Oct 30 22:02:35.302736 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:35.302711    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-jsqpx"
Oct 30 22:02:35.302997 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:35.302972    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-jsqpx"
Oct 30 22:02:35.303643 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:35.303624    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-jsqpx"
Oct 30 22:02:40.571460 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:02:40.571421    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 22:02:55.919681 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:55.919644    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-755df48df5-dhkgj"]
Oct 30 22:02:55.925626 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:55.925604    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-755df48df5-dhkgj"
Oct 30 22:02:55.982832 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:55.982802    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-755df48df5-dhkgj"]
Oct 30 22:02:56.027797 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:56.027765    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-sf4vq\""
Oct 30 22:02:56.027797 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:56.027786    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\""
Oct 30 22:02:56.027797 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:56.027788    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\""
Oct 30 22:02:56.029208 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:56.029186    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vzk5\" (UniqueName: \"kubernetes.io/projected/87daad87-329d-4840-a2e8-61fd6f03beac-kube-api-access-6vzk5\") pod \"ai-gateway-controller-755df48df5-dhkgj\" (UID: \"87daad87-329d-4840-a2e8-61fd6f03beac\") " pod="opendatahub/ai-gateway-controller-755df48df5-dhkgj"
Oct 30 22:02:56.029287 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:56.029257    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87daad87-329d-4840-a2e8-61fd6f03beac-metrics-certs\") pod \"ai-gateway-controller-755df48df5-dhkgj\" (UID: \"87daad87-329d-4840-a2e8-61fd6f03beac\") " pod="opendatahub/ai-gateway-controller-755df48df5-dhkgj"
Oct 30 22:02:56.130104 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:56.130061    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vzk5\" (UniqueName: \"kubernetes.io/projected/87daad87-329d-4840-a2e8-61fd6f03beac-kube-api-access-6vzk5\") pod \"ai-gateway-controller-755df48df5-dhkgj\" (UID: \"87daad87-329d-4840-a2e8-61fd6f03beac\") " pod="opendatahub/ai-gateway-controller-755df48df5-dhkgj"
Oct 30 22:02:56.130302 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:56.130152    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87daad87-329d-4840-a2e8-61fd6f03beac-metrics-certs\") pod \"ai-gateway-controller-755df48df5-dhkgj\" (UID: \"87daad87-329d-4840-a2e8-61fd6f03beac\") " pod="opendatahub/ai-gateway-controller-755df48df5-dhkgj"
Oct 30 22:02:56.132618 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:56.132594    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87daad87-329d-4840-a2e8-61fd6f03beac-metrics-certs\") pod \"ai-gateway-controller-755df48df5-dhkgj\" (UID: \"87daad87-329d-4840-a2e8-61fd6f03beac\") " pod="opendatahub/ai-gateway-controller-755df48df5-dhkgj"
Oct 30 22:02:56.188644 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:56.188580    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vzk5\" (UniqueName: \"kubernetes.io/projected/87daad87-329d-4840-a2e8-61fd6f03beac-kube-api-access-6vzk5\") pod \"ai-gateway-controller-755df48df5-dhkgj\" (UID: \"87daad87-329d-4840-a2e8-61fd6f03beac\") " pod="opendatahub/ai-gateway-controller-755df48df5-dhkgj"
Oct 30 22:02:56.237337 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:56.237311    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-755df48df5-dhkgj"
Oct 30 22:02:56.611249 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:56.611225    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-755df48df5-dhkgj"]
Oct 30 22:02:56.613868 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:02:56.613831    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87daad87_329d_4840_a2e8_61fd6f03beac.slice/crio-f9d2c0da42f9b4280d51d58760900bf9f67880495b36cdbcb285be50e44f1b7a WatchSource:0}: Error finding container f9d2c0da42f9b4280d51d58760900bf9f67880495b36cdbcb285be50e44f1b7a: Status 404 returned error can't find the container with id f9d2c0da42f9b4280d51d58760900bf9f67880495b36cdbcb285be50e44f1b7a
Oct 30 22:02:56.723059 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:56.723023    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-755df48df5-dhkgj" event={"ID":"87daad87-329d-4840-a2e8-61fd6f03beac","Type":"ContainerStarted","Data":"f9d2c0da42f9b4280d51d58760900bf9f67880495b36cdbcb285be50e44f1b7a"}
Oct 30 22:02:59.739089 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:59.739053    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-755df48df5-dhkgj" event={"ID":"87daad87-329d-4840-a2e8-61fd6f03beac","Type":"ContainerStarted","Data":"9483e5a25632fdfd8ff31e98933ca81cbd56ac5ed35cd43aff3589b664aac4a6"}
Oct 30 22:02:59.739502 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:59.739139    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-755df48df5-dhkgj"
Oct 30 22:02:59.851015 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:02:59.850912    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-755df48df5-dhkgj" podStartSLOduration=1.9624184169999999 podStartE2EDuration="4.850897402s" totalImagesPullingTime="2.888478985s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:02:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:02:59.849697475 +0000 UTC m=+757.454005030" watchObservedRunningTime="2026-10-30 22:02:59.850897402 +0000 UTC m=+757.455204954"
Oct 30 22:03:10.572396 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:03:10.572357    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 22:03:10.746776 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:10.746747    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-755df48df5-dhkgj"
Oct 30 22:03:21.804930 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:21.804894    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-qjw2b"]
Oct 30 22:03:21.807801 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:21.807781    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-qjw2b"
Oct 30 22:03:21.852292 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:21.852257    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d9083c7-76bf-4db9-be88-f8045b5d3c77-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-qjw2b\" (UID: \"5d9083c7-76bf-4db9-be88-f8045b5d3c77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-qjw2b"
Oct 30 22:03:21.852292 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:21.852293    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d9083c7-76bf-4db9-be88-f8045b5d3c77-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-qjw2b\" (UID: \"5d9083c7-76bf-4db9-be88-f8045b5d3c77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-qjw2b"
Oct 30 22:03:21.852587 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:21.852318    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d9083c7-76bf-4db9-be88-f8045b5d3c77-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-qjw2b\" (UID: \"5d9083c7-76bf-4db9-be88-f8045b5d3c77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-qjw2b"
Oct 30 22:03:21.852587 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:21.852366    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlzd8\" (UniqueName: \"kubernetes.io/projected/5d9083c7-76bf-4db9-be88-f8045b5d3c77-kube-api-access-tlzd8\") pod \"maas-default-gateway-openshift-default-85544d4cd7-qjw2b\" (UID: \"5d9083c7-76bf-4db9-be88-f8045b5d3c77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-qjw2b"
Oct 30 22:03:21.852587 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:21.852458    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d9083c7-76bf-4db9-be88-f8045b5d3c77-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-qjw2b\" (UID: \"5d9083c7-76bf-4db9-be88-f8045b5d3c77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-qjw2b"
Oct 30 22:03:21.852587 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:21.852484    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d9083c7-76bf-4db9-be88-f8045b5d3c77-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-qjw2b\" (UID: \"5d9083c7-76bf-4db9-be88-f8045b5d3c77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-qjw2b"
Oct 30 22:03:21.852587 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:21.852521    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d9083c7-76bf-4db9-be88-f8045b5d3c77-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-qjw2b\" (UID: \"5d9083c7-76bf-4db9-be88-f8045b5d3c77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-qjw2b"
Oct 30 22:03:21.852814 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:21.852657    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d9083c7-76bf-4db9-be88-f8045b5d3c77-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-qjw2b\" (UID: \"5d9083c7-76bf-4db9-be88-f8045b5d3c77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-qjw2b"
Oct 30 22:03:21.852814 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:21.852688    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d9083c7-76bf-4db9-be88-f8045b5d3c77-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-qjw2b\" (UID: \"5d9083c7-76bf-4db9-be88-f8045b5d3c77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-qjw2b"
Oct 30 22:03:21.922450 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:21.922414    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-qjw2b"]
Oct 30 22:03:21.953954 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:21.953924    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d9083c7-76bf-4db9-be88-f8045b5d3c77-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-qjw2b\" (UID: \"5d9083c7-76bf-4db9-be88-f8045b5d3c77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-qjw2b"
Oct 30 22:03:21.953954 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:21.953958    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d9083c7-76bf-4db9-be88-f8045b5d3c77-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-qjw2b\" (UID: \"5d9083c7-76bf-4db9-be88-f8045b5d3c77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-qjw2b"
Oct 30 22:03:21.954192 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:21.953999    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d9083c7-76bf-4db9-be88-f8045b5d3c77-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-qjw2b\" (UID: \"5d9083c7-76bf-4db9-be88-f8045b5d3c77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-qjw2b"
Oct 30 22:03:21.954192 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:21.954027    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d9083c7-76bf-4db9-be88-f8045b5d3c77-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-qjw2b\" (UID: \"5d9083c7-76bf-4db9-be88-f8045b5d3c77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-qjw2b"
Oct 30 22:03:21.954192 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:21.954069    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d9083c7-76bf-4db9-be88-f8045b5d3c77-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-qjw2b\" (UID: \"5d9083c7-76bf-4db9-be88-f8045b5d3c77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-qjw2b"
Oct 30 22:03:21.954192 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:21.954091    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlzd8\" (UniqueName: \"kubernetes.io/projected/5d9083c7-76bf-4db9-be88-f8045b5d3c77-kube-api-access-tlzd8\") pod \"maas-default-gateway-openshift-default-85544d4cd7-qjw2b\" (UID: \"5d9083c7-76bf-4db9-be88-f8045b5d3c77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-qjw2b"
Oct 30 22:03:21.954192 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:21.954127    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d9083c7-76bf-4db9-be88-f8045b5d3c77-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-qjw2b\" (UID: \"5d9083c7-76bf-4db9-be88-f8045b5d3c77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-qjw2b"
Oct 30 22:03:21.954192 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:21.954160    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d9083c7-76bf-4db9-be88-f8045b5d3c77-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-qjw2b\" (UID: \"5d9083c7-76bf-4db9-be88-f8045b5d3c77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-qjw2b"
Oct 30 22:03:21.954529 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:21.954224    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d9083c7-76bf-4db9-be88-f8045b5d3c77-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-qjw2b\" (UID: \"5d9083c7-76bf-4db9-be88-f8045b5d3c77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-qjw2b"
Oct 30 22:03:21.954585 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:21.954548    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d9083c7-76bf-4db9-be88-f8045b5d3c77-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-qjw2b\" (UID: \"5d9083c7-76bf-4db9-be88-f8045b5d3c77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-qjw2b"
Oct 30 22:03:21.954637 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:21.954603    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d9083c7-76bf-4db9-be88-f8045b5d3c77-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-qjw2b\" (UID: \"5d9083c7-76bf-4db9-be88-f8045b5d3c77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-qjw2b"
Oct 30 22:03:21.954732 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:21.954707    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d9083c7-76bf-4db9-be88-f8045b5d3c77-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-qjw2b\" (UID: \"5d9083c7-76bf-4db9-be88-f8045b5d3c77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-qjw2b"
Oct 30 22:03:21.954907 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:21.954873    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d9083c7-76bf-4db9-be88-f8045b5d3c77-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-qjw2b\" (UID: \"5d9083c7-76bf-4db9-be88-f8045b5d3c77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-qjw2b"
Oct 30 22:03:21.955148 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:21.955125    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d9083c7-76bf-4db9-be88-f8045b5d3c77-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-qjw2b\" (UID: \"5d9083c7-76bf-4db9-be88-f8045b5d3c77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-qjw2b"
Oct 30 22:03:21.956921 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:21.956897    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d9083c7-76bf-4db9-be88-f8045b5d3c77-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-qjw2b\" (UID: \"5d9083c7-76bf-4db9-be88-f8045b5d3c77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-qjw2b"
Oct 30 22:03:21.957475 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:21.957453    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d9083c7-76bf-4db9-be88-f8045b5d3c77-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-qjw2b\" (UID: \"5d9083c7-76bf-4db9-be88-f8045b5d3c77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-qjw2b"
Oct 30 22:03:22.014229 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:22.014200    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlzd8\" (UniqueName: \"kubernetes.io/projected/5d9083c7-76bf-4db9-be88-f8045b5d3c77-kube-api-access-tlzd8\") pod \"maas-default-gateway-openshift-default-85544d4cd7-qjw2b\" (UID: \"5d9083c7-76bf-4db9-be88-f8045b5d3c77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-qjw2b"
Oct 30 22:03:22.015734 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:22.015709    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d9083c7-76bf-4db9-be88-f8045b5d3c77-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-qjw2b\" (UID: \"5d9083c7-76bf-4db9-be88-f8045b5d3c77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-qjw2b"
Oct 30 22:03:22.120358 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:22.120321    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-qjw2b"
Oct 30 22:03:22.550732 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:22.550711    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-qjw2b"]
Oct 30 22:03:22.552801 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:03:22.552762    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d9083c7_76bf_4db9_be88_f8045b5d3c77.slice/crio-dc1aa9c0b849afea337d4aaf0a1496b5c8ba58324dc949790cec1c51149b3136 WatchSource:0}: Error finding container dc1aa9c0b849afea337d4aaf0a1496b5c8ba58324dc949790cec1c51149b3136: Status 404 returned error can't find the container with id dc1aa9c0b849afea337d4aaf0a1496b5c8ba58324dc949790cec1c51149b3136
Oct 30 22:03:22.555456 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:22.555425    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:03:22.555564 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:22.555490    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:03:22.555564 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:22.555518    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:03:22.844071 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:22.843986    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-qjw2b" event={"ID":"5d9083c7-76bf-4db9-be88-f8045b5d3c77","Type":"ContainerStarted","Data":"27f781697fae8b6a60da646656bf76cfa4fb21de2a9deddff449594215ceecdf"}
Oct 30 22:03:22.844071 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:22.844040    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-qjw2b" event={"ID":"5d9083c7-76bf-4db9-be88-f8045b5d3c77","Type":"ContainerStarted","Data":"dc1aa9c0b849afea337d4aaf0a1496b5c8ba58324dc949790cec1c51149b3136"}
Oct 30 22:03:22.961272 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:22.961217    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-qjw2b" podStartSLOduration=1.961204009 podStartE2EDuration="1.961204009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:03:22.959124287 +0000 UTC m=+780.563431841" watchObservedRunningTime="2026-10-30 22:03:22.961204009 +0000 UTC m=+780.565511562"
Oct 30 22:03:23.120583 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:23.120528    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-qjw2b"
Oct 30 22:03:23.145573 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:23.145533    5678 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-qjw2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Oct 30 22:03:23.145841 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:23.145807    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-qjw2b" podUID="5d9083c7-76bf-4db9-be88-f8045b5d3c77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Oct 30 22:03:24.120732 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:24.120700    5678 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-qjw2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Oct 30 22:03:24.121118 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:24.120750    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-qjw2b" podUID="5d9083c7-76bf-4db9-be88-f8045b5d3c77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Oct 30 22:03:25.121032 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:25.120997    5678 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-qjw2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Oct 30 22:03:25.121434 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:25.121057    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-qjw2b" podUID="5d9083c7-76bf-4db9-be88-f8045b5d3c77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Oct 30 22:03:26.121520 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:26.121485    5678 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-qjw2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Oct 30 22:03:26.133535 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:26.133493    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-qjw2b" podUID="5d9083c7-76bf-4db9-be88-f8045b5d3c77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Oct 30 22:03:27.130429 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:27.121440    5678 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-qjw2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Oct 30 22:03:27.130429 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:27.121521    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-qjw2b" podUID="5d9083c7-76bf-4db9-be88-f8045b5d3c77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Oct 30 22:03:28.120834 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:28.120790    5678 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-qjw2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Oct 30 22:03:28.121155 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:28.120853    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-qjw2b" podUID="5d9083c7-76bf-4db9-be88-f8045b5d3c77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Oct 30 22:03:29.121129 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:29.121100    5678 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-qjw2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Oct 30 22:03:29.121540 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:29.121148    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-qjw2b" podUID="5d9083c7-76bf-4db9-be88-f8045b5d3c77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Oct 30 22:03:30.755668 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:30.755633    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-qjw2b"
Oct 30 22:03:30.756841 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:30.756775    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-qjw2b"
Oct 30 22:03:30.757604 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:30.757581    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-qjw2b"
Oct 30 22:03:40.571209 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:03:40.571174    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 22:03:42.275096 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:42.275061    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cdbcf"]
Oct 30 22:03:42.277803 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:42.277781    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cdbcf"
Oct 30 22:03:42.336187 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:42.336157    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-779f32-openshift-default-dockercfg-lp8gt\""
Oct 30 22:03:42.337566 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:42.337528    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4618a35f-6775-4be2-ab02-e5c9940e13b0-workload-certs\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cdbcf\" (UID: \"4618a35f-6775-4be2-ab02-e5c9940e13b0\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cdbcf"
Oct 30 22:03:42.337728 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:42.337584    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4618a35f-6775-4be2-ab02-e5c9940e13b0-istio-token\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cdbcf\" (UID: \"4618a35f-6775-4be2-ab02-e5c9940e13b0\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cdbcf"
Oct 30 22:03:42.337790 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:42.337711    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4618a35f-6775-4be2-ab02-e5c9940e13b0-istio-podinfo\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cdbcf\" (UID: \"4618a35f-6775-4be2-ab02-e5c9940e13b0\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cdbcf"
Oct 30 22:03:42.337790 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:42.337764    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4618a35f-6775-4be2-ab02-e5c9940e13b0-credential-socket\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cdbcf\" (UID: \"4618a35f-6775-4be2-ab02-e5c9940e13b0\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cdbcf"
Oct 30 22:03:42.337900 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:42.337834    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4618a35f-6775-4be2-ab02-e5c9940e13b0-istio-data\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cdbcf\" (UID: \"4618a35f-6775-4be2-ab02-e5c9940e13b0\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cdbcf"
Oct 30 22:03:42.337900 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:42.337865    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9c9gs\" (UniqueName: \"kubernetes.io/projected/4618a35f-6775-4be2-ab02-e5c9940e13b0-kube-api-access-9c9gs\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cdbcf\" (UID: \"4618a35f-6775-4be2-ab02-e5c9940e13b0\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cdbcf"
Oct 30 22:03:42.338001 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:42.337901    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4618a35f-6775-4be2-ab02-e5c9940e13b0-istio-envoy\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cdbcf\" (UID: \"4618a35f-6775-4be2-ab02-e5c9940e13b0\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cdbcf"
Oct 30 22:03:42.338001 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:42.337938    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4618a35f-6775-4be2-ab02-e5c9940e13b0-workload-socket\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cdbcf\" (UID: \"4618a35f-6775-4be2-ab02-e5c9940e13b0\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cdbcf"
Oct 30 22:03:42.338101 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:42.338039    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4618a35f-6775-4be2-ab02-e5c9940e13b0-istiod-ca-cert\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cdbcf\" (UID: \"4618a35f-6775-4be2-ab02-e5c9940e13b0\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cdbcf"
Oct 30 22:03:42.382222 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:42.382193    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cdbcf"]
Oct 30 22:03:42.438850 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:42.438817    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4618a35f-6775-4be2-ab02-e5c9940e13b0-istiod-ca-cert\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cdbcf\" (UID: \"4618a35f-6775-4be2-ab02-e5c9940e13b0\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cdbcf"
Oct 30 22:03:42.438850 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:42.438855    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4618a35f-6775-4be2-ab02-e5c9940e13b0-workload-certs\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cdbcf\" (UID: \"4618a35f-6775-4be2-ab02-e5c9940e13b0\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cdbcf"
Oct 30 22:03:42.439058 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:42.438879    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4618a35f-6775-4be2-ab02-e5c9940e13b0-istio-token\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cdbcf\" (UID: \"4618a35f-6775-4be2-ab02-e5c9940e13b0\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cdbcf"
Oct 30 22:03:42.439058 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:42.438904    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4618a35f-6775-4be2-ab02-e5c9940e13b0-istio-podinfo\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cdbcf\" (UID: \"4618a35f-6775-4be2-ab02-e5c9940e13b0\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cdbcf"
Oct 30 22:03:42.439058 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:42.438927    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4618a35f-6775-4be2-ab02-e5c9940e13b0-credential-socket\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cdbcf\" (UID: \"4618a35f-6775-4be2-ab02-e5c9940e13b0\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cdbcf"
Oct 30 22:03:42.439058 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:42.438946    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4618a35f-6775-4be2-ab02-e5c9940e13b0-istio-data\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cdbcf\" (UID: \"4618a35f-6775-4be2-ab02-e5c9940e13b0\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cdbcf"
Oct 30 22:03:42.439058 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:42.438965    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9c9gs\" (UniqueName: \"kubernetes.io/projected/4618a35f-6775-4be2-ab02-e5c9940e13b0-kube-api-access-9c9gs\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cdbcf\" (UID: \"4618a35f-6775-4be2-ab02-e5c9940e13b0\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cdbcf"
Oct 30 22:03:42.439058 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:42.438999    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4618a35f-6775-4be2-ab02-e5c9940e13b0-istio-envoy\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cdbcf\" (UID: \"4618a35f-6775-4be2-ab02-e5c9940e13b0\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cdbcf"
Oct 30 22:03:42.439058 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:42.439028    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4618a35f-6775-4be2-ab02-e5c9940e13b0-workload-socket\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cdbcf\" (UID: \"4618a35f-6775-4be2-ab02-e5c9940e13b0\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cdbcf"
Oct 30 22:03:42.439678 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:42.439649    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4618a35f-6775-4be2-ab02-e5c9940e13b0-credential-socket\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cdbcf\" (UID: \"4618a35f-6775-4be2-ab02-e5c9940e13b0\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cdbcf"
Oct 30 22:03:42.439795 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:42.439756    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4618a35f-6775-4be2-ab02-e5c9940e13b0-istio-data\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cdbcf\" (UID: \"4618a35f-6775-4be2-ab02-e5c9940e13b0\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cdbcf"
Oct 30 22:03:42.439996 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:42.439965    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4618a35f-6775-4be2-ab02-e5c9940e13b0-workload-certs\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cdbcf\" (UID: \"4618a35f-6775-4be2-ab02-e5c9940e13b0\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cdbcf"
Oct 30 22:03:42.440139 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:42.440120    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4618a35f-6775-4be2-ab02-e5c9940e13b0-workload-socket\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cdbcf\" (UID: \"4618a35f-6775-4be2-ab02-e5c9940e13b0\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cdbcf"
Oct 30 22:03:42.442458 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:42.440350    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4618a35f-6775-4be2-ab02-e5c9940e13b0-istiod-ca-cert\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cdbcf\" (UID: \"4618a35f-6775-4be2-ab02-e5c9940e13b0\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cdbcf"
Oct 30 22:03:42.444425 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:42.443380    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4618a35f-6775-4be2-ab02-e5c9940e13b0-istio-envoy\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cdbcf\" (UID: \"4618a35f-6775-4be2-ab02-e5c9940e13b0\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cdbcf"
Oct 30 22:03:42.447700 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:42.447679    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4618a35f-6775-4be2-ab02-e5c9940e13b0-istio-podinfo\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cdbcf\" (UID: \"4618a35f-6775-4be2-ab02-e5c9940e13b0\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cdbcf"
Oct 30 22:03:42.500723 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:42.500688    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9c9gs\" (UniqueName: \"kubernetes.io/projected/4618a35f-6775-4be2-ab02-e5c9940e13b0-kube-api-access-9c9gs\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cdbcf\" (UID: \"4618a35f-6775-4be2-ab02-e5c9940e13b0\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cdbcf"
Oct 30 22:03:42.506630 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:42.506599    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4618a35f-6775-4be2-ab02-e5c9940e13b0-istio-token\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cdbcf\" (UID: \"4618a35f-6775-4be2-ab02-e5c9940e13b0\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cdbcf"
Oct 30 22:03:42.591340 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:42.591267    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cdbcf"
Oct 30 22:03:42.952997 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:42.952963    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-wqqzd"]
Oct 30 22:03:42.956649 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:42.956623    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-wqqzd"
Oct 30 22:03:43.006620 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:43.006595    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-bc714d-openshift-default-dockercfg-gs4pf\""
Oct 30 22:03:43.020959 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:43.020933    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-wqqzd"]
Oct 30 22:03:43.031681 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:43.031606    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cdbcf"]
Oct 30 22:03:43.033908 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:03:43.033879    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4618a35f_6775_4be2_ab02_e5c9940e13b0.slice/crio-51b0b5b7343ef4a4f1579a876a08bfb83870d8ff522ddf526d14314b84cdc43c WatchSource:0}: Error finding container 51b0b5b7343ef4a4f1579a876a08bfb83870d8ff522ddf526d14314b84cdc43c: Status 404 returned error can't find the container with id 51b0b5b7343ef4a4f1579a876a08bfb83870d8ff522ddf526d14314b84cdc43c
Oct 30 22:03:43.036266 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:43.036238    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:03:43.036326 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:43.036298    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:03:43.036362 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:43.036325    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:03:43.049290 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:43.049265    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99f99481-73d0-4fb4-9c88-12ee143a6904-istio-podinfo\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-wqqzd\" (UID: \"99f99481-73d0-4fb4-9c88-12ee143a6904\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-wqqzd"
Oct 30 22:03:43.049386 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:43.049301    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99f99481-73d0-4fb4-9c88-12ee143a6904-istiod-ca-cert\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-wqqzd\" (UID: \"99f99481-73d0-4fb4-9c88-12ee143a6904\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-wqqzd"
Oct 30 22:03:43.049386 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:43.049323    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99f99481-73d0-4fb4-9c88-12ee143a6904-istio-envoy\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-wqqzd\" (UID: \"99f99481-73d0-4fb4-9c88-12ee143a6904\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-wqqzd"
Oct 30 22:03:43.049386 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:43.049376    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s95sf\" (UniqueName: \"kubernetes.io/projected/99f99481-73d0-4fb4-9c88-12ee143a6904-kube-api-access-s95sf\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-wqqzd\" (UID: \"99f99481-73d0-4fb4-9c88-12ee143a6904\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-wqqzd"
Oct 30 22:03:43.049613 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:43.049583    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99f99481-73d0-4fb4-9c88-12ee143a6904-istio-data\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-wqqzd\" (UID: \"99f99481-73d0-4fb4-9c88-12ee143a6904\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-wqqzd"
Oct 30 22:03:43.049706 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:43.049653    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99f99481-73d0-4fb4-9c88-12ee143a6904-istio-token\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-wqqzd\" (UID: \"99f99481-73d0-4fb4-9c88-12ee143a6904\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-wqqzd"
Oct 30 22:03:43.049706 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:43.049675    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99f99481-73d0-4fb4-9c88-12ee143a6904-workload-certs\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-wqqzd\" (UID: \"99f99481-73d0-4fb4-9c88-12ee143a6904\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-wqqzd"
Oct 30 22:03:43.049785 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:43.049704    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99f99481-73d0-4fb4-9c88-12ee143a6904-credential-socket\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-wqqzd\" (UID: \"99f99481-73d0-4fb4-9c88-12ee143a6904\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-wqqzd"
Oct 30 22:03:43.049870 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:43.049846    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99f99481-73d0-4fb4-9c88-12ee143a6904-workload-socket\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-wqqzd\" (UID: \"99f99481-73d0-4fb4-9c88-12ee143a6904\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-wqqzd"
Oct 30 22:03:43.151153 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:43.151117    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99f99481-73d0-4fb4-9c88-12ee143a6904-credential-socket\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-wqqzd\" (UID: \"99f99481-73d0-4fb4-9c88-12ee143a6904\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-wqqzd"
Oct 30 22:03:43.151340 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:43.151226    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99f99481-73d0-4fb4-9c88-12ee143a6904-workload-socket\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-wqqzd\" (UID: \"99f99481-73d0-4fb4-9c88-12ee143a6904\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-wqqzd"
Oct 30 22:03:43.151340 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:43.151269    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99f99481-73d0-4fb4-9c88-12ee143a6904-istio-podinfo\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-wqqzd\" (UID: \"99f99481-73d0-4fb4-9c88-12ee143a6904\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-wqqzd"
Oct 30 22:03:43.151340 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:43.151296    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99f99481-73d0-4fb4-9c88-12ee143a6904-istiod-ca-cert\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-wqqzd\" (UID: \"99f99481-73d0-4fb4-9c88-12ee143a6904\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-wqqzd"
Oct 30 22:03:43.151562 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:43.151434    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99f99481-73d0-4fb4-9c88-12ee143a6904-istio-envoy\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-wqqzd\" (UID: \"99f99481-73d0-4fb4-9c88-12ee143a6904\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-wqqzd"
Oct 30 22:03:43.151562 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:43.151492    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s95sf\" (UniqueName: \"kubernetes.io/projected/99f99481-73d0-4fb4-9c88-12ee143a6904-kube-api-access-s95sf\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-wqqzd\" (UID: \"99f99481-73d0-4fb4-9c88-12ee143a6904\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-wqqzd"
Oct 30 22:03:43.151562 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:43.151527    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99f99481-73d0-4fb4-9c88-12ee143a6904-istio-data\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-wqqzd\" (UID: \"99f99481-73d0-4fb4-9c88-12ee143a6904\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-wqqzd"
Oct 30 22:03:43.151728 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:43.151566    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99f99481-73d0-4fb4-9c88-12ee143a6904-istio-token\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-wqqzd\" (UID: \"99f99481-73d0-4fb4-9c88-12ee143a6904\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-wqqzd"
Oct 30 22:03:43.151728 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:43.151590    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99f99481-73d0-4fb4-9c88-12ee143a6904-workload-certs\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-wqqzd\" (UID: \"99f99481-73d0-4fb4-9c88-12ee143a6904\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-wqqzd"
Oct 30 22:03:43.151728 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:43.151604    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99f99481-73d0-4fb4-9c88-12ee143a6904-credential-socket\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-wqqzd\" (UID: \"99f99481-73d0-4fb4-9c88-12ee143a6904\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-wqqzd"
Oct 30 22:03:43.151728 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:43.151639    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99f99481-73d0-4fb4-9c88-12ee143a6904-workload-socket\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-wqqzd\" (UID: \"99f99481-73d0-4fb4-9c88-12ee143a6904\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-wqqzd"
Oct 30 22:03:43.151958 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:43.151884    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99f99481-73d0-4fb4-9c88-12ee143a6904-workload-certs\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-wqqzd\" (UID: \"99f99481-73d0-4fb4-9c88-12ee143a6904\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-wqqzd"
Oct 30 22:03:43.152092 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:43.152074    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99f99481-73d0-4fb4-9c88-12ee143a6904-istiod-ca-cert\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-wqqzd\" (UID: \"99f99481-73d0-4fb4-9c88-12ee143a6904\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-wqqzd"
Oct 30 22:03:43.152158 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:43.152115    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99f99481-73d0-4fb4-9c88-12ee143a6904-istio-data\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-wqqzd\" (UID: \"99f99481-73d0-4fb4-9c88-12ee143a6904\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-wqqzd"
Oct 30 22:03:43.154051 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:43.154030    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99f99481-73d0-4fb4-9c88-12ee143a6904-istio-podinfo\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-wqqzd\" (UID: \"99f99481-73d0-4fb4-9c88-12ee143a6904\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-wqqzd"
Oct 30 22:03:43.154179 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:43.154162    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99f99481-73d0-4fb4-9c88-12ee143a6904-istio-envoy\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-wqqzd\" (UID: \"99f99481-73d0-4fb4-9c88-12ee143a6904\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-wqqzd"
Oct 30 22:03:43.213549 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:43.213477    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99f99481-73d0-4fb4-9c88-12ee143a6904-istio-token\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-wqqzd\" (UID: \"99f99481-73d0-4fb4-9c88-12ee143a6904\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-wqqzd"
Oct 30 22:03:43.213840 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:43.213819    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s95sf\" (UniqueName: \"kubernetes.io/projected/99f99481-73d0-4fb4-9c88-12ee143a6904-kube-api-access-s95sf\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-wqqzd\" (UID: \"99f99481-73d0-4fb4-9c88-12ee143a6904\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-wqqzd"
Oct 30 22:03:43.269319 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:43.269291    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-wqqzd"
Oct 30 22:03:43.683833 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:43.683803    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9fln8"]
Oct 30 22:03:43.687609 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:43.687583    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9fln8"
Oct 30 22:03:43.736131 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:43.736104    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-wqqzd"]
Oct 30 22:03:43.737601 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:03:43.737571    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99f99481_73d0_4fb4_9c88_12ee143a6904.slice/crio-f3df1686d80b7438a2cbbe728098857646e59cc5768b12b047bd47917b87389e WatchSource:0}: Error finding container f3df1686d80b7438a2cbbe728098857646e59cc5768b12b047bd47917b87389e: Status 404 returned error can't find the container with id f3df1686d80b7438a2cbbe728098857646e59cc5768b12b047bd47917b87389e
Oct 30 22:03:43.740093 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:43.740061    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:03:43.740219 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:43.740142    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:03:43.740219 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:43.740187    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:03:43.751884 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:43.751866    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-a00d9b-openshift-default-dockercfg-w7t4n\""
Oct 30 22:03:43.757514 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:43.757487    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jc8zs\" (UniqueName: \"kubernetes.io/projected/dbbbcda0-3462-4c73-af87-c1c8fcca4706-kube-api-access-jc8zs\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9fln8\" (UID: \"dbbbcda0-3462-4c73-af87-c1c8fcca4706\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9fln8"
Oct 30 22:03:43.757620 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:43.757537    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbbbcda0-3462-4c73-af87-c1c8fcca4706-istio-envoy\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9fln8\" (UID: \"dbbbcda0-3462-4c73-af87-c1c8fcca4706\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9fln8"
Oct 30 22:03:43.757620 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:43.757577    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbbbcda0-3462-4c73-af87-c1c8fcca4706-istio-data\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9fln8\" (UID: \"dbbbcda0-3462-4c73-af87-c1c8fcca4706\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9fln8"
Oct 30 22:03:43.757743 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:43.757640    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbbbcda0-3462-4c73-af87-c1c8fcca4706-credential-socket\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9fln8\" (UID: \"dbbbcda0-3462-4c73-af87-c1c8fcca4706\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9fln8"
Oct 30 22:03:43.757743 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:43.757681    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbbbcda0-3462-4c73-af87-c1c8fcca4706-workload-certs\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9fln8\" (UID: \"dbbbcda0-3462-4c73-af87-c1c8fcca4706\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9fln8"
Oct 30 22:03:43.757743 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:43.757715    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbbbcda0-3462-4c73-af87-c1c8fcca4706-istio-token\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9fln8\" (UID: \"dbbbcda0-3462-4c73-af87-c1c8fcca4706\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9fln8"
Oct 30 22:03:43.757903 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:43.757756    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbbbcda0-3462-4c73-af87-c1c8fcca4706-istio-podinfo\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9fln8\" (UID: \"dbbbcda0-3462-4c73-af87-c1c8fcca4706\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9fln8"
Oct 30 22:03:43.757903 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:43.757788    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbbbcda0-3462-4c73-af87-c1c8fcca4706-workload-socket\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9fln8\" (UID: \"dbbbcda0-3462-4c73-af87-c1c8fcca4706\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9fln8"
Oct 30 22:03:43.757903 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:43.757841    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbbbcda0-3462-4c73-af87-c1c8fcca4706-istiod-ca-cert\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9fln8\" (UID: \"dbbbcda0-3462-4c73-af87-c1c8fcca4706\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9fln8"
Oct 30 22:03:43.760088 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:43.760066    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9fln8"]
Oct 30 22:03:43.858784 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:43.858736    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jc8zs\" (UniqueName: \"kubernetes.io/projected/dbbbcda0-3462-4c73-af87-c1c8fcca4706-kube-api-access-jc8zs\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9fln8\" (UID: \"dbbbcda0-3462-4c73-af87-c1c8fcca4706\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9fln8"
Oct 30 22:03:43.858985 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:43.858793    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbbbcda0-3462-4c73-af87-c1c8fcca4706-istio-envoy\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9fln8\" (UID: \"dbbbcda0-3462-4c73-af87-c1c8fcca4706\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9fln8"
Oct 30 22:03:43.858985 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:43.858831    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbbbcda0-3462-4c73-af87-c1c8fcca4706-istio-data\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9fln8\" (UID: \"dbbbcda0-3462-4c73-af87-c1c8fcca4706\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9fln8"
Oct 30 22:03:43.858985 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:43.858864    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbbbcda0-3462-4c73-af87-c1c8fcca4706-credential-socket\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9fln8\" (UID: \"dbbbcda0-3462-4c73-af87-c1c8fcca4706\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9fln8"
Oct 30 22:03:43.858985 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:43.858897    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbbbcda0-3462-4c73-af87-c1c8fcca4706-workload-certs\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9fln8\" (UID: \"dbbbcda0-3462-4c73-af87-c1c8fcca4706\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9fln8"
Oct 30 22:03:43.859203 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:43.859003    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbbbcda0-3462-4c73-af87-c1c8fcca4706-istio-token\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9fln8\" (UID: \"dbbbcda0-3462-4c73-af87-c1c8fcca4706\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9fln8"
Oct 30 22:03:43.859203 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:43.859058    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbbbcda0-3462-4c73-af87-c1c8fcca4706-istio-podinfo\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9fln8\" (UID: \"dbbbcda0-3462-4c73-af87-c1c8fcca4706\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9fln8"
Oct 30 22:03:43.859203 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:43.859094    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbbbcda0-3462-4c73-af87-c1c8fcca4706-workload-socket\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9fln8\" (UID: \"dbbbcda0-3462-4c73-af87-c1c8fcca4706\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9fln8"
Oct 30 22:03:43.859203 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:43.859152    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbbbcda0-3462-4c73-af87-c1c8fcca4706-istiod-ca-cert\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9fln8\" (UID: \"dbbbcda0-3462-4c73-af87-c1c8fcca4706\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9fln8"
Oct 30 22:03:43.859424 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:43.859306    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbbbcda0-3462-4c73-af87-c1c8fcca4706-credential-socket\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9fln8\" (UID: \"dbbbcda0-3462-4c73-af87-c1c8fcca4706\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9fln8"
Oct 30 22:03:43.859424 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:43.859336    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbbbcda0-3462-4c73-af87-c1c8fcca4706-workload-certs\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9fln8\" (UID: \"dbbbcda0-3462-4c73-af87-c1c8fcca4706\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9fln8"
Oct 30 22:03:43.859619 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:43.859590    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbbbcda0-3462-4c73-af87-c1c8fcca4706-workload-socket\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9fln8\" (UID: \"dbbbcda0-3462-4c73-af87-c1c8fcca4706\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9fln8"
Oct 30 22:03:43.859842 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:43.859822    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbbbcda0-3462-4c73-af87-c1c8fcca4706-istiod-ca-cert\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9fln8\" (UID: \"dbbbcda0-3462-4c73-af87-c1c8fcca4706\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9fln8"
Oct 30 22:03:43.860054 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:43.860036    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbbbcda0-3462-4c73-af87-c1c8fcca4706-istio-data\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9fln8\" (UID: \"dbbbcda0-3462-4c73-af87-c1c8fcca4706\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9fln8"
Oct 30 22:03:43.861535 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:43.861510    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbbbcda0-3462-4c73-af87-c1c8fcca4706-istio-podinfo\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9fln8\" (UID: \"dbbbcda0-3462-4c73-af87-c1c8fcca4706\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9fln8"
Oct 30 22:03:43.861640 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:43.861535    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbbbcda0-3462-4c73-af87-c1c8fcca4706-istio-envoy\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9fln8\" (UID: \"dbbbcda0-3462-4c73-af87-c1c8fcca4706\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9fln8"
Oct 30 22:03:43.917681 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:43.917645    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbbbcda0-3462-4c73-af87-c1c8fcca4706-istio-token\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9fln8\" (UID: \"dbbbcda0-3462-4c73-af87-c1c8fcca4706\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9fln8"
Oct 30 22:03:43.917823 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:43.917773    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jc8zs\" (UniqueName: \"kubernetes.io/projected/dbbbcda0-3462-4c73-af87-c1c8fcca4706-kube-api-access-jc8zs\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9fln8\" (UID: \"dbbbcda0-3462-4c73-af87-c1c8fcca4706\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9fln8"
Oct 30 22:03:43.944862 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:43.944788    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-wqqzd" event={"ID":"99f99481-73d0-4fb4-9c88-12ee143a6904","Type":"ContainerStarted","Data":"1850898ccd8dc9a1d5251132a95b3f1e312f10352d5a5baaea7fc97ccbe83434"}
Oct 30 22:03:43.944862 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:43.944852    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-wqqzd" event={"ID":"99f99481-73d0-4fb4-9c88-12ee143a6904","Type":"ContainerStarted","Data":"f3df1686d80b7438a2cbbe728098857646e59cc5768b12b047bd47917b87389e"}
Oct 30 22:03:43.946410 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:43.946369    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cdbcf" event={"ID":"4618a35f-6775-4be2-ab02-e5c9940e13b0","Type":"ContainerStarted","Data":"656ee0b1af412dbc9cc40e18f123f9e2dc69a6ac2770b6536e90afe20e196612"}
Oct 30 22:03:43.946526 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:43.946437    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cdbcf" event={"ID":"4618a35f-6775-4be2-ab02-e5c9940e13b0","Type":"ContainerStarted","Data":"51b0b5b7343ef4a4f1579a876a08bfb83870d8ff522ddf526d14314b84cdc43c"}
Oct 30 22:03:44.004529 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:44.004484    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9fln8"
Oct 30 22:03:44.075724 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:44.071364    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-wqqzd" podStartSLOduration=2.071351022 podStartE2EDuration="2.071351022s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:03:44.067650146 +0000 UTC m=+801.671957705" watchObservedRunningTime="2026-10-30 22:03:44.071351022 +0000 UTC m=+801.675658575"
Oct 30 22:03:44.186573 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:44.186512    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cdbcf" podStartSLOduration=2.186494569 podStartE2EDuration="2.186494569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:03:44.183334274 +0000 UTC m=+801.787641827" watchObservedRunningTime="2026-10-30 22:03:44.186494569 +0000 UTC m=+801.790802121"
Oct 30 22:03:44.270004 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:44.269925    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-wqqzd"
Oct 30 22:03:44.274738 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:44.274712    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-wqqzd"
Oct 30 22:03:44.401215 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:44.401181    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-qp9q4"]
Oct 30 22:03:44.408763 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:44.408585    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-qp9q4"
Oct 30 22:03:44.456043 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:44.456017    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9fln8"]
Oct 30 22:03:44.458779 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:03:44.458726    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbbbcda0_3462_4c73_af87_c1c8fcca4706.slice/crio-a6eb2f676a5cedcff70bbb53f7b35e1eb4b1e3f8a26e059c68b2c96fdab9df85 WatchSource:0}: Error finding container a6eb2f676a5cedcff70bbb53f7b35e1eb4b1e3f8a26e059c68b2c96fdab9df85: Status 404 returned error can't find the container with id a6eb2f676a5cedcff70bbb53f7b35e1eb4b1e3f8a26e059c68b2c96fdab9df85
Oct 30 22:03:44.461257 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:44.461226    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:03:44.461344 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:44.461286    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:03:44.461344 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:44.461316    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:03:44.463532 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:44.463512    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-fb0b93-openshift-default-dockercfg-2kctd\""
Oct 30 22:03:44.466548 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:44.465330    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwn6c\" (UniqueName: \"kubernetes.io/projected/e400e450-98f2-423b-bbc1-478fb88f61a2-kube-api-access-xwn6c\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-qp9q4\" (UID: \"e400e450-98f2-423b-bbc1-478fb88f61a2\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-qp9q4"
Oct 30 22:03:44.466548 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:44.465386    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e400e450-98f2-423b-bbc1-478fb88f61a2-istio-data\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-qp9q4\" (UID: \"e400e450-98f2-423b-bbc1-478fb88f61a2\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-qp9q4"
Oct 30 22:03:44.466548 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:44.465495    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e400e450-98f2-423b-bbc1-478fb88f61a2-istio-podinfo\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-qp9q4\" (UID: \"e400e450-98f2-423b-bbc1-478fb88f61a2\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-qp9q4"
Oct 30 22:03:44.466548 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:44.465566    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e400e450-98f2-423b-bbc1-478fb88f61a2-istio-token\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-qp9q4\" (UID: \"e400e450-98f2-423b-bbc1-478fb88f61a2\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-qp9q4"
Oct 30 22:03:44.466548 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:44.465621    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e400e450-98f2-423b-bbc1-478fb88f61a2-istio-envoy\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-qp9q4\" (UID: \"e400e450-98f2-423b-bbc1-478fb88f61a2\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-qp9q4"
Oct 30 22:03:44.466548 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:44.465675    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e400e450-98f2-423b-bbc1-478fb88f61a2-credential-socket\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-qp9q4\" (UID: \"e400e450-98f2-423b-bbc1-478fb88f61a2\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-qp9q4"
Oct 30 22:03:44.466548 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:44.465698    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e400e450-98f2-423b-bbc1-478fb88f61a2-workload-certs\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-qp9q4\" (UID: \"e400e450-98f2-423b-bbc1-478fb88f61a2\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-qp9q4"
Oct 30 22:03:44.466548 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:44.465752    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e400e450-98f2-423b-bbc1-478fb88f61a2-workload-socket\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-qp9q4\" (UID: \"e400e450-98f2-423b-bbc1-478fb88f61a2\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-qp9q4"
Oct 30 22:03:44.466548 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:44.465837    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e400e450-98f2-423b-bbc1-478fb88f61a2-istiod-ca-cert\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-qp9q4\" (UID: \"e400e450-98f2-423b-bbc1-478fb88f61a2\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-qp9q4"
Oct 30 22:03:44.466548 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:44.466264    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-qp9q4"]
Oct 30 22:03:44.566594 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:44.566562    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e400e450-98f2-423b-bbc1-478fb88f61a2-istio-data\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-qp9q4\" (UID: \"e400e450-98f2-423b-bbc1-478fb88f61a2\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-qp9q4"
Oct 30 22:03:44.566773 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:44.566601    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e400e450-98f2-423b-bbc1-478fb88f61a2-istio-podinfo\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-qp9q4\" (UID: \"e400e450-98f2-423b-bbc1-478fb88f61a2\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-qp9q4"
Oct 30 22:03:44.566773 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:44.566630    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e400e450-98f2-423b-bbc1-478fb88f61a2-istio-token\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-qp9q4\" (UID: \"e400e450-98f2-423b-bbc1-478fb88f61a2\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-qp9q4"
Oct 30 22:03:44.566773 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:44.566657    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e400e450-98f2-423b-bbc1-478fb88f61a2-istio-envoy\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-qp9q4\" (UID: \"e400e450-98f2-423b-bbc1-478fb88f61a2\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-qp9q4"
Oct 30 22:03:44.566773 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:44.566711    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e400e450-98f2-423b-bbc1-478fb88f61a2-credential-socket\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-qp9q4\" (UID: \"e400e450-98f2-423b-bbc1-478fb88f61a2\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-qp9q4"
Oct 30 22:03:44.566773 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:44.566735    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e400e450-98f2-423b-bbc1-478fb88f61a2-workload-certs\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-qp9q4\" (UID: \"e400e450-98f2-423b-bbc1-478fb88f61a2\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-qp9q4"
Oct 30 22:03:44.567035 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:44.566794    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e400e450-98f2-423b-bbc1-478fb88f61a2-workload-socket\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-qp9q4\" (UID: \"e400e450-98f2-423b-bbc1-478fb88f61a2\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-qp9q4"
Oct 30 22:03:44.567035 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:44.566909    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e400e450-98f2-423b-bbc1-478fb88f61a2-istiod-ca-cert\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-qp9q4\" (UID: \"e400e450-98f2-423b-bbc1-478fb88f61a2\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-qp9q4"
Oct 30 22:03:44.567035 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:44.566971    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwn6c\" (UniqueName: \"kubernetes.io/projected/e400e450-98f2-423b-bbc1-478fb88f61a2-kube-api-access-xwn6c\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-qp9q4\" (UID: \"e400e450-98f2-423b-bbc1-478fb88f61a2\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-qp9q4"
Oct 30 22:03:44.567035 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:44.566992    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e400e450-98f2-423b-bbc1-478fb88f61a2-istio-data\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-qp9q4\" (UID: \"e400e450-98f2-423b-bbc1-478fb88f61a2\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-qp9q4"
Oct 30 22:03:44.567254 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:44.567128    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e400e450-98f2-423b-bbc1-478fb88f61a2-credential-socket\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-qp9q4\" (UID: \"e400e450-98f2-423b-bbc1-478fb88f61a2\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-qp9q4"
Oct 30 22:03:44.567254 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:44.567221    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e400e450-98f2-423b-bbc1-478fb88f61a2-workload-socket\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-qp9q4\" (UID: \"e400e450-98f2-423b-bbc1-478fb88f61a2\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-qp9q4"
Oct 30 22:03:44.567664 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:44.567640    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e400e450-98f2-423b-bbc1-478fb88f61a2-istiod-ca-cert\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-qp9q4\" (UID: \"e400e450-98f2-423b-bbc1-478fb88f61a2\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-qp9q4"
Oct 30 22:03:44.567664 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:44.567660    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e400e450-98f2-423b-bbc1-478fb88f61a2-workload-certs\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-qp9q4\" (UID: \"e400e450-98f2-423b-bbc1-478fb88f61a2\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-qp9q4"
Oct 30 22:03:44.569361 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:44.569329    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e400e450-98f2-423b-bbc1-478fb88f61a2-istio-podinfo\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-qp9q4\" (UID: \"e400e450-98f2-423b-bbc1-478fb88f61a2\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-qp9q4"
Oct 30 22:03:44.569607 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:44.569588    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e400e450-98f2-423b-bbc1-478fb88f61a2-istio-envoy\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-qp9q4\" (UID: \"e400e450-98f2-423b-bbc1-478fb88f61a2\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-qp9q4"
Oct 30 22:03:44.591631 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:44.591612    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cdbcf"
Oct 30 22:03:44.596514 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:44.596484    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cdbcf"
Oct 30 22:03:44.624717 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:44.624696    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e400e450-98f2-423b-bbc1-478fb88f61a2-istio-token\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-qp9q4\" (UID: \"e400e450-98f2-423b-bbc1-478fb88f61a2\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-qp9q4"
Oct 30 22:03:44.626261 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:44.626237    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwn6c\" (UniqueName: \"kubernetes.io/projected/e400e450-98f2-423b-bbc1-478fb88f61a2-kube-api-access-xwn6c\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-qp9q4\" (UID: \"e400e450-98f2-423b-bbc1-478fb88f61a2\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-qp9q4"
Oct 30 22:03:44.734067 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:44.734032    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-qp9q4"
Oct 30 22:03:44.953215 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:44.953177    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9fln8" event={"ID":"dbbbcda0-3462-4c73-af87-c1c8fcca4706","Type":"ContainerStarted","Data":"495ad04d6a89274ae44c70c17d1fc9defd3d924808919743235cb5ea283530fb"}
Oct 30 22:03:44.953417 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:44.953230    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9fln8" event={"ID":"dbbbcda0-3462-4c73-af87-c1c8fcca4706","Type":"ContainerStarted","Data":"a6eb2f676a5cedcff70bbb53f7b35e1eb4b1e3f8a26e059c68b2c96fdab9df85"}
Oct 30 22:03:44.953699 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:44.953673    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cdbcf"
Oct 30 22:03:44.953812 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:44.953726    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-wqqzd"
Oct 30 22:03:44.954304 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:44.954282    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cdbcf"
Oct 30 22:03:44.954385 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:44.954333    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-wqqzd"
Oct 30 22:03:45.005305 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:45.005280    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9fln8"
Oct 30 22:03:45.006815 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:45.006793    5678 patch_prober.go:29] interesting pod/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9fln8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Oct 30 22:03:45.006965 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:45.006837    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9fln8" podUID="dbbbcda0-3462-4c73-af87-c1c8fcca4706" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Oct 30 22:03:45.071963 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:45.071868    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9fln8" podStartSLOduration=2.071855152 podStartE2EDuration="2.071855152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:03:45.069291018 +0000 UTC m=+802.673598599" watchObservedRunningTime="2026-10-30 22:03:45.071855152 +0000 UTC m=+802.676162772"
Oct 30 22:03:45.189491 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:45.189466    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-qp9q4"]
Oct 30 22:03:45.207862 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:45.207810    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:03:45.208034 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:45.207930    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:03:45.208034 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:45.207998    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:03:45.965652 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:45.965615    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-qp9q4" event={"ID":"e400e450-98f2-423b-bbc1-478fb88f61a2","Type":"ContainerStarted","Data":"6b480b2933e1c4531fb1444aed1d873b1593d1a10905c1b7420b8291ce7ada3c"}
Oct 30 22:03:45.966047 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:45.965669    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-qp9q4" event={"ID":"e400e450-98f2-423b-bbc1-478fb88f61a2","Type":"ContainerStarted","Data":"c7cfb1978eca7aea1fd6da84e51952f9c2afd520ea1fde298e0e176a0296e0ac"}
Oct 30 22:03:46.009013 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:46.008982    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9fln8"
Oct 30 22:03:46.082195 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:46.082131    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-qp9q4" podStartSLOduration=2.082117482 podStartE2EDuration="2.082117482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:03:46.080172475 +0000 UTC m=+803.684480038" watchObservedRunningTime="2026-10-30 22:03:46.082117482 +0000 UTC m=+803.686425035"
Oct 30 22:03:46.735078 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:46.735048    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-qp9q4"
Oct 30 22:03:46.740484 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:46.740458    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-qp9q4"
Oct 30 22:03:46.978926 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:46.978881    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9fln8"
Oct 30 22:03:46.979390 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:46.978968    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-qp9q4"
Oct 30 22:03:46.979462 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:46.979429    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-qp9q4"
Oct 30 22:03:46.979569 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:46.979550    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9fln8"
Oct 30 22:03:56.301092 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:56.301061    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-bc714d-57765d6599-txzbm"]
Oct 30 22:03:56.304680 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:56.304658    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-bc714d-57765d6599-txzbm"
Oct 30 22:03:56.359258 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:56.359229    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-bc714d-57765d6599-txzbm"]
Oct 30 22:03:56.385036 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:56.385003    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b45b1a6-24c2-4170-b914-85c2a9a5aab8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-bc714d-57765d6599-txzbm\" (UID: \"6b45b1a6-24c2-4170-b914-85c2a9a5aab8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-bc714d-57765d6599-txzbm"
Oct 30 22:03:56.385191 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:56.385065    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gm97\" (UniqueName: \"kubernetes.io/projected/6b45b1a6-24c2-4170-b914-85c2a9a5aab8-kube-api-access-8gm97\") pod \"payload-pre-processing-e2e-worker-w2-bc714d-57765d6599-txzbm\" (UID: \"6b45b1a6-24c2-4170-b914-85c2a9a5aab8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-bc714d-57765d6599-txzbm"
Oct 30 22:03:56.403131 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:56.403106    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-bc714d\""
Oct 30 22:03:56.403131 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:56.403106    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-bc714d-dockercfg-228nv\""
Oct 30 22:03:56.458896 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:56.458866    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-bc714d-7c5b5598f-phvl6"]
Oct 30 22:03:56.461673 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:56.461647    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-bc714d-7c5b5598f-phvl6"
Oct 30 22:03:56.486263 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:56.486231    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b45b1a6-24c2-4170-b914-85c2a9a5aab8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-bc714d-57765d6599-txzbm\" (UID: \"6b45b1a6-24c2-4170-b914-85c2a9a5aab8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-bc714d-57765d6599-txzbm"
Oct 30 22:03:56.486441 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:56.486301    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gm97\" (UniqueName: \"kubernetes.io/projected/6b45b1a6-24c2-4170-b914-85c2a9a5aab8-kube-api-access-8gm97\") pod \"payload-pre-processing-e2e-worker-w2-bc714d-57765d6599-txzbm\" (UID: \"6b45b1a6-24c2-4170-b914-85c2a9a5aab8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-bc714d-57765d6599-txzbm"
Oct 30 22:03:56.487048 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:56.487022    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b45b1a6-24c2-4170-b914-85c2a9a5aab8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-bc714d-57765d6599-txzbm\" (UID: \"6b45b1a6-24c2-4170-b914-85c2a9a5aab8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-bc714d-57765d6599-txzbm"
Oct 30 22:03:56.525130 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:56.525096    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-bc714d-7c5b5598f-phvl6"]
Oct 30 22:03:56.545174 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:56.545143    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gm97\" (UniqueName: \"kubernetes.io/projected/6b45b1a6-24c2-4170-b914-85c2a9a5aab8-kube-api-access-8gm97\") pod \"payload-pre-processing-e2e-worker-w2-bc714d-57765d6599-txzbm\" (UID: \"6b45b1a6-24c2-4170-b914-85c2a9a5aab8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-bc714d-57765d6599-txzbm"
Oct 30 22:03:56.588004 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:56.587929    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdn7q\" (UniqueName: \"kubernetes.io/projected/5f9fdabd-6c3b-4bbe-b1ea-ef2159076fc1-kube-api-access-kdn7q\") pod \"payload-processing-e2e-worker-w2-bc714d-7c5b5598f-phvl6\" (UID: \"5f9fdabd-6c3b-4bbe-b1ea-ef2159076fc1\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-bc714d-7c5b5598f-phvl6"
Oct 30 22:03:56.588139 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:56.588120    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f9fdabd-6c3b-4bbe-b1ea-ef2159076fc1-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-bc714d-7c5b5598f-phvl6\" (UID: \"5f9fdabd-6c3b-4bbe-b1ea-ef2159076fc1\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-bc714d-7c5b5598f-phvl6"
Oct 30 22:03:56.648106 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:56.648076    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-bc714d-57765d6599-txzbm"
Oct 30 22:03:56.689686 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:56.689637    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f9fdabd-6c3b-4bbe-b1ea-ef2159076fc1-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-bc714d-7c5b5598f-phvl6\" (UID: \"5f9fdabd-6c3b-4bbe-b1ea-ef2159076fc1\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-bc714d-7c5b5598f-phvl6"
Oct 30 22:03:56.689823 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:56.689735    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdn7q\" (UniqueName: \"kubernetes.io/projected/5f9fdabd-6c3b-4bbe-b1ea-ef2159076fc1-kube-api-access-kdn7q\") pod \"payload-processing-e2e-worker-w2-bc714d-7c5b5598f-phvl6\" (UID: \"5f9fdabd-6c3b-4bbe-b1ea-ef2159076fc1\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-bc714d-7c5b5598f-phvl6"
Oct 30 22:03:56.690575 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:56.690552    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f9fdabd-6c3b-4bbe-b1ea-ef2159076fc1-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-bc714d-7c5b5598f-phvl6\" (UID: \"5f9fdabd-6c3b-4bbe-b1ea-ef2159076fc1\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-bc714d-7c5b5598f-phvl6"
Oct 30 22:03:56.748576 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:56.748550    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdn7q\" (UniqueName: \"kubernetes.io/projected/5f9fdabd-6c3b-4bbe-b1ea-ef2159076fc1-kube-api-access-kdn7q\") pod \"payload-processing-e2e-worker-w2-bc714d-7c5b5598f-phvl6\" (UID: \"5f9fdabd-6c3b-4bbe-b1ea-ef2159076fc1\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-bc714d-7c5b5598f-phvl6"
Oct 30 22:03:56.777325 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:56.777297    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-bc714d-7c5b5598f-phvl6"
Oct 30 22:03:57.022032 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:57.022006    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-bc714d-57765d6599-txzbm"]
Oct 30 22:03:57.024239 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:03:57.024210    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b45b1a6_24c2_4170_b914_85c2a9a5aab8.slice/crio-6dbcac47238d3ec8b6313a06671609af6b69078e327f7ba0535cbe8eb44d5d01 WatchSource:0}: Error finding container 6dbcac47238d3ec8b6313a06671609af6b69078e327f7ba0535cbe8eb44d5d01: Status 404 returned error can't find the container with id 6dbcac47238d3ec8b6313a06671609af6b69078e327f7ba0535cbe8eb44d5d01
Oct 30 22:03:57.031683 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:57.031655    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-bc714d-57765d6599-txzbm" event={"ID":"6b45b1a6-24c2-4170-b914-85c2a9a5aab8","Type":"ContainerStarted","Data":"6dbcac47238d3ec8b6313a06671609af6b69078e327f7ba0535cbe8eb44d5d01"}
Oct 30 22:03:57.155183 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:57.155160    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-bc714d-7c5b5598f-phvl6"]
Oct 30 22:03:57.156881 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:03:57.156858    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f9fdabd_6c3b_4bbe_b1ea_ef2159076fc1.slice/crio-59c0dfd9b78853b2ad679a444401d735783b2813e727b32f75bbbfe70c51d180 WatchSource:0}: Error finding container 59c0dfd9b78853b2ad679a444401d735783b2813e727b32f75bbbfe70c51d180: Status 404 returned error can't find the container with id 59c0dfd9b78853b2ad679a444401d735783b2813e727b32f75bbbfe70c51d180
Oct 30 22:03:58.037426 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:58.037370    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-bc714d-7c5b5598f-phvl6" event={"ID":"5f9fdabd-6c3b-4bbe-b1ea-ef2159076fc1","Type":"ContainerStarted","Data":"59c0dfd9b78853b2ad679a444401d735783b2813e727b32f75bbbfe70c51d180"}
Oct 30 22:03:59.045220 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:59.043473    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-779f32-66c9bd7459-g2qwk"]
Oct 30 22:03:59.047848 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:59.047819    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-779f32-66c9bd7459-g2qwk"
Oct 30 22:03:59.144899 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:59.144861    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-779f32-dockercfg-d6dj7\""
Oct 30 22:03:59.146130 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:59.146089    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-779f32\""
Oct 30 22:03:59.148099 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:59.148076    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-779f32-66c9bd7459-g2qwk"]
Oct 30 22:03:59.217459 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:59.217426    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdeffce0-5736-4ac8-89b1-cfdd262fd5c7-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-779f32-66c9bd7459-g2qwk\" (UID: \"bdeffce0-5736-4ac8-89b1-cfdd262fd5c7\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-779f32-66c9bd7459-g2qwk"
Oct 30 22:03:59.217618 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:59.217531    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tn7pf\" (UniqueName: \"kubernetes.io/projected/bdeffce0-5736-4ac8-89b1-cfdd262fd5c7-kube-api-access-tn7pf\") pod \"payload-processing-e2e-worker-w0-779f32-66c9bd7459-g2qwk\" (UID: \"bdeffce0-5736-4ac8-89b1-cfdd262fd5c7\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-779f32-66c9bd7459-g2qwk"
Oct 30 22:03:59.319197 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:59.318665    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdeffce0-5736-4ac8-89b1-cfdd262fd5c7-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-779f32-66c9bd7459-g2qwk\" (UID: \"bdeffce0-5736-4ac8-89b1-cfdd262fd5c7\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-779f32-66c9bd7459-g2qwk"
Oct 30 22:03:59.319197 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:59.318773    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tn7pf\" (UniqueName: \"kubernetes.io/projected/bdeffce0-5736-4ac8-89b1-cfdd262fd5c7-kube-api-access-tn7pf\") pod \"payload-processing-e2e-worker-w0-779f32-66c9bd7459-g2qwk\" (UID: \"bdeffce0-5736-4ac8-89b1-cfdd262fd5c7\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-779f32-66c9bd7459-g2qwk"
Oct 30 22:03:59.319631 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:59.319606    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdeffce0-5736-4ac8-89b1-cfdd262fd5c7-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-779f32-66c9bd7459-g2qwk\" (UID: \"bdeffce0-5736-4ac8-89b1-cfdd262fd5c7\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-779f32-66c9bd7459-g2qwk"
Oct 30 22:03:59.388516 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:59.380836    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tn7pf\" (UniqueName: \"kubernetes.io/projected/bdeffce0-5736-4ac8-89b1-cfdd262fd5c7-kube-api-access-tn7pf\") pod \"payload-processing-e2e-worker-w0-779f32-66c9bd7459-g2qwk\" (UID: \"bdeffce0-5736-4ac8-89b1-cfdd262fd5c7\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-779f32-66c9bd7459-g2qwk"
Oct 30 22:03:59.663510 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:03:59.663469    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-779f32-66c9bd7459-g2qwk"
Oct 30 22:04:00.054479 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:00.054378    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-779f32-66c9bd7459-g2qwk"]
Oct 30 22:04:01.057635 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:01.057602    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-bc714d-57765d6599-txzbm" event={"ID":"6b45b1a6-24c2-4170-b914-85c2a9a5aab8","Type":"ContainerStarted","Data":"5078d8146efb238945600b2bd45ccd3ef4bd2815da0a9fe04112eefa833904c0"}
Oct 30 22:04:01.057995 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:01.057715    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-bc714d-57765d6599-txzbm"
Oct 30 22:04:01.059334 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:01.059308    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-bc714d-7c5b5598f-phvl6" event={"ID":"5f9fdabd-6c3b-4bbe-b1ea-ef2159076fc1","Type":"ContainerStarted","Data":"fd76d1901f436997d6f834a701e64501c1344ff94f4b4466632872e5b535dc00"}
Oct 30 22:04:01.059470 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:01.059359    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-bc714d-7c5b5598f-phvl6"
Oct 30 22:04:01.060776 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:01.060755    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-779f32-66c9bd7459-g2qwk" event={"ID":"bdeffce0-5736-4ac8-89b1-cfdd262fd5c7","Type":"ContainerStarted","Data":"4fbc28e64b4ec94e9f0c6ae2007d7f3b7541fbd8b23721a36dfdf378c87e576e"}
Oct 30 22:04:01.060901 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:01.060793    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-779f32-66c9bd7459-g2qwk"
Oct 30 22:04:01.060901 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:01.060809    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-779f32-66c9bd7459-g2qwk" event={"ID":"bdeffce0-5736-4ac8-89b1-cfdd262fd5c7","Type":"ContainerStarted","Data":"9912a36c3edd9d447daff888c071264964a6f8532ef6e25b90a3096fcfaeddd1"}
Oct 30 22:04:01.170696 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:01.170602    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-bc714d-57765d6599-txzbm" podStartSLOduration=1.977077076 podStartE2EDuration="5.170588754s" totalImagesPullingTime="3.193511678s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:01.16825252 +0000 UTC m=+818.772560073" watchObservedRunningTime="2026-10-30 22:04:01.170588754 +0000 UTC m=+818.774896307"
Oct 30 22:04:01.283902 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:01.283795    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-bc714d-7c5b5598f-phvl6" podStartSLOduration=2.064493997 podStartE2EDuration="5.283781078s" totalImagesPullingTime="3.219287081s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:01.282147743 +0000 UTC m=+818.886455295" watchObservedRunningTime="2026-10-30 22:04:01.283781078 +0000 UTC m=+818.888088649"
Oct 30 22:04:01.394677 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:01.394621    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-779f32-66c9bd7459-g2qwk" podStartSLOduration=2.472554821 podStartE2EDuration="3.394606796s" totalImagesPullingTime="922.051975ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:01.392953333 +0000 UTC m=+818.997260888" watchObservedRunningTime="2026-10-30 22:04:01.394606796 +0000 UTC m=+818.998914348"
Oct 30 22:04:01.563218 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:01.563140    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-a00d9b-55d8cbb68c-dvsfc"]
Oct 30 22:04:01.566272 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:01.566252    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a00d9b-55d8cbb68c-dvsfc"
Oct 30 22:04:01.615809 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:01.615783    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-a00d9b\""
Oct 30 22:04:01.615918 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:01.615812    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-a00d9b-dockercfg-9d5qh\""
Oct 30 22:04:01.621448 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:01.621426    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-a00d9b-55d8cbb68c-dvsfc"]
Oct 30 22:04:01.745501 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:01.745461    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vn99n\" (UniqueName: \"kubernetes.io/projected/a2f7629d-e9a0-438b-b4b0-b33d5b6a47cb-kube-api-access-vn99n\") pod \"payload-pre-processing-e2e-worker-w1-a00d9b-55d8cbb68c-dvsfc\" (UID: \"a2f7629d-e9a0-438b-b4b0-b33d5b6a47cb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a00d9b-55d8cbb68c-dvsfc"
Oct 30 22:04:01.745664 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:01.745519    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2f7629d-e9a0-438b-b4b0-b33d5b6a47cb-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-a00d9b-55d8cbb68c-dvsfc\" (UID: \"a2f7629d-e9a0-438b-b4b0-b33d5b6a47cb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a00d9b-55d8cbb68c-dvsfc"
Oct 30 22:04:01.846324 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:01.846228    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vn99n\" (UniqueName: \"kubernetes.io/projected/a2f7629d-e9a0-438b-b4b0-b33d5b6a47cb-kube-api-access-vn99n\") pod \"payload-pre-processing-e2e-worker-w1-a00d9b-55d8cbb68c-dvsfc\" (UID: \"a2f7629d-e9a0-438b-b4b0-b33d5b6a47cb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a00d9b-55d8cbb68c-dvsfc"
Oct 30 22:04:01.846324 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:01.846282    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2f7629d-e9a0-438b-b4b0-b33d5b6a47cb-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-a00d9b-55d8cbb68c-dvsfc\" (UID: \"a2f7629d-e9a0-438b-b4b0-b33d5b6a47cb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a00d9b-55d8cbb68c-dvsfc"
Oct 30 22:04:01.847040 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:01.847018    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2f7629d-e9a0-438b-b4b0-b33d5b6a47cb-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-a00d9b-55d8cbb68c-dvsfc\" (UID: \"a2f7629d-e9a0-438b-b4b0-b33d5b6a47cb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a00d9b-55d8cbb68c-dvsfc"
Oct 30 22:04:01.905237 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:01.905201    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vn99n\" (UniqueName: \"kubernetes.io/projected/a2f7629d-e9a0-438b-b4b0-b33d5b6a47cb-kube-api-access-vn99n\") pod \"payload-pre-processing-e2e-worker-w1-a00d9b-55d8cbb68c-dvsfc\" (UID: \"a2f7629d-e9a0-438b-b4b0-b33d5b6a47cb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a00d9b-55d8cbb68c-dvsfc"
Oct 30 22:04:02.179228 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:02.179199    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a00d9b-55d8cbb68c-dvsfc"
Oct 30 22:04:02.554055 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:02.554017    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-a00d9b-55d8cbb68c-dvsfc"]
Oct 30 22:04:02.557496 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:04:02.557461    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2f7629d_e9a0_438b_b4b0_b33d5b6a47cb.slice/crio-e85d4ae8649d3a77b9562ca67583814273a434c9c60b19ec7223f3394cfcb0d2 WatchSource:0}: Error finding container e85d4ae8649d3a77b9562ca67583814273a434c9c60b19ec7223f3394cfcb0d2: Status 404 returned error can't find the container with id e85d4ae8649d3a77b9562ca67583814273a434c9c60b19ec7223f3394cfcb0d2
Oct 30 22:04:03.074451 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:03.074413    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a00d9b-55d8cbb68c-dvsfc" event={"ID":"a2f7629d-e9a0-438b-b4b0-b33d5b6a47cb","Type":"ContainerStarted","Data":"215376cb1f32dd4a31c29cc482b7bd8680821c0cff40a6089f3fc7c2c9354150"}
Oct 30 22:04:03.074619 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:03.074466    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a00d9b-55d8cbb68c-dvsfc" event={"ID":"a2f7629d-e9a0-438b-b4b0-b33d5b6a47cb","Type":"ContainerStarted","Data":"e85d4ae8649d3a77b9562ca67583814273a434c9c60b19ec7223f3394cfcb0d2"}
Oct 30 22:04:03.074664 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:03.074623    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a00d9b-55d8cbb68c-dvsfc"
Oct 30 22:04:03.248480 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:03.247251    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a00d9b-55d8cbb68c-dvsfc" podStartSLOduration=2.247221667 podStartE2EDuration="2.247221667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:03.228893353 +0000 UTC m=+820.833200929" watchObservedRunningTime="2026-10-30 22:04:03.247221667 +0000 UTC m=+820.851529223"
Oct 30 22:04:06.999457 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:06.999425    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-65e59f-74d8d9b4c-t64kq"]
Oct 30 22:04:07.002713 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:07.002687    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-65e59f-74d8d9b4c-t64kq"
Oct 30 22:04:07.054032 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:07.053999    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-65e59f-dockercfg-pgbkn\""
Oct 30 22:04:07.100951 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:07.100927    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-65e59f\""
Oct 30 22:04:07.103319 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:07.103298    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-65e59f-74d8d9b4c-t64kq"]
Oct 30 22:04:07.103694 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:07.103674    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlrxg\" (UniqueName: \"kubernetes.io/projected/4b088ba1-b952-4a98-9316-7673c67ad3c8-kube-api-access-qlrxg\") pod \"payload-pre-processing-e2e-worker-w4-65e59f-74d8d9b4c-t64kq\" (UID: \"4b088ba1-b952-4a98-9316-7673c67ad3c8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-65e59f-74d8d9b4c-t64kq"
Oct 30 22:04:07.103828 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:07.103813    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b088ba1-b952-4a98-9316-7673c67ad3c8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-65e59f-74d8d9b4c-t64kq\" (UID: \"4b088ba1-b952-4a98-9316-7673c67ad3c8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-65e59f-74d8d9b4c-t64kq"
Oct 30 22:04:07.205377 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:07.205334    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b088ba1-b952-4a98-9316-7673c67ad3c8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-65e59f-74d8d9b4c-t64kq\" (UID: \"4b088ba1-b952-4a98-9316-7673c67ad3c8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-65e59f-74d8d9b4c-t64kq"
Oct 30 22:04:07.205581 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:07.205492    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlrxg\" (UniqueName: \"kubernetes.io/projected/4b088ba1-b952-4a98-9316-7673c67ad3c8-kube-api-access-qlrxg\") pod \"payload-pre-processing-e2e-worker-w4-65e59f-74d8d9b4c-t64kq\" (UID: \"4b088ba1-b952-4a98-9316-7673c67ad3c8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-65e59f-74d8d9b4c-t64kq"
Oct 30 22:04:07.206189 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:07.206160    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b088ba1-b952-4a98-9316-7673c67ad3c8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-65e59f-74d8d9b4c-t64kq\" (UID: \"4b088ba1-b952-4a98-9316-7673c67ad3c8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-65e59f-74d8d9b4c-t64kq"
Oct 30 22:04:07.265174 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:07.265088    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlrxg\" (UniqueName: \"kubernetes.io/projected/4b088ba1-b952-4a98-9316-7673c67ad3c8-kube-api-access-qlrxg\") pod \"payload-pre-processing-e2e-worker-w4-65e59f-74d8d9b4c-t64kq\" (UID: \"4b088ba1-b952-4a98-9316-7673c67ad3c8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-65e59f-74d8d9b4c-t64kq"
Oct 30 22:04:07.315040 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:07.315003    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-65e59f-74d8d9b4c-t64kq"
Oct 30 22:04:07.716699 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:07.716677    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-65e59f-74d8d9b4c-t64kq"]
Oct 30 22:04:07.718870 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:04:07.718840    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b088ba1_b952_4a98_9316_7673c67ad3c8.slice/crio-7655793ef295df75503fd9705e83b77986f31ba5d23b7b09f81bac92089119db WatchSource:0}: Error finding container 7655793ef295df75503fd9705e83b77986f31ba5d23b7b09f81bac92089119db: Status 404 returned error can't find the container with id 7655793ef295df75503fd9705e83b77986f31ba5d23b7b09f81bac92089119db
Oct 30 22:04:08.117487 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:08.117447    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-65e59f-74d8d9b4c-t64kq" event={"ID":"4b088ba1-b952-4a98-9316-7673c67ad3c8","Type":"ContainerStarted","Data":"8d7f8b4b344dbd3851a421b5cabafb83cdc35454617ae8b7c82b8e720979bcda"}
Oct 30 22:04:08.118028 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:08.117514    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-65e59f-74d8d9b4c-t64kq" event={"ID":"4b088ba1-b952-4a98-9316-7673c67ad3c8","Type":"ContainerStarted","Data":"7655793ef295df75503fd9705e83b77986f31ba5d23b7b09f81bac92089119db"}
Oct 30 22:04:08.119345 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:08.118514    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-65e59f-74d8d9b4c-t64kq"
Oct 30 22:04:08.234473 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:08.234420    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-65e59f-74d8d9b4c-t64kq" podStartSLOduration=2.234383343 podStartE2EDuration="2.234383343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:08.232771911 +0000 UTC m=+825.837079487" watchObservedRunningTime="2026-10-30 22:04:08.234383343 +0000 UTC m=+825.838690896"
Oct 30 22:04:09.698490 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:09.694148    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-fb0b93-7fb666d8f9-wr46f"]
Oct 30 22:04:09.699481 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:09.699453    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-fb0b93-7fb666d8f9-wr46f"
Oct 30 22:04:09.730344 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:09.730310    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24066223-6a66-43ef-a85d-83183bc37fa7-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-fb0b93-7fb666d8f9-wr46f\" (UID: \"24066223-6a66-43ef-a85d-83183bc37fa7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-fb0b93-7fb666d8f9-wr46f"
Oct 30 22:04:09.730541 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:09.730361    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdrcd\" (UniqueName: \"kubernetes.io/projected/24066223-6a66-43ef-a85d-83183bc37fa7-kube-api-access-tdrcd\") pod \"payload-pre-processing-e2e-worker-w6-fb0b93-7fb666d8f9-wr46f\" (UID: \"24066223-6a66-43ef-a85d-83183bc37fa7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-fb0b93-7fb666d8f9-wr46f"
Oct 30 22:04:09.753770 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:09.753737    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-fb0b93-7fb666d8f9-wr46f"]
Oct 30 22:04:09.796885 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:09.796857    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-fb0b93\""
Oct 30 22:04:09.831350 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:09.831313    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24066223-6a66-43ef-a85d-83183bc37fa7-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-fb0b93-7fb666d8f9-wr46f\" (UID: \"24066223-6a66-43ef-a85d-83183bc37fa7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-fb0b93-7fb666d8f9-wr46f"
Oct 30 22:04:09.831544 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:09.831365    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdrcd\" (UniqueName: \"kubernetes.io/projected/24066223-6a66-43ef-a85d-83183bc37fa7-kube-api-access-tdrcd\") pod \"payload-pre-processing-e2e-worker-w6-fb0b93-7fb666d8f9-wr46f\" (UID: \"24066223-6a66-43ef-a85d-83183bc37fa7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-fb0b93-7fb666d8f9-wr46f"
Oct 30 22:04:09.832066 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:09.832046    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24066223-6a66-43ef-a85d-83183bc37fa7-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-fb0b93-7fb666d8f9-wr46f\" (UID: \"24066223-6a66-43ef-a85d-83183bc37fa7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-fb0b93-7fb666d8f9-wr46f"
Oct 30 22:04:09.890631 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:09.890598    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdrcd\" (UniqueName: \"kubernetes.io/projected/24066223-6a66-43ef-a85d-83183bc37fa7-kube-api-access-tdrcd\") pod \"payload-pre-processing-e2e-worker-w6-fb0b93-7fb666d8f9-wr46f\" (UID: \"24066223-6a66-43ef-a85d-83183bc37fa7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-fb0b93-7fb666d8f9-wr46f"
Oct 30 22:04:10.023718 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:10.023641    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-fb0b93-7fb666d8f9-wr46f"
Oct 30 22:04:10.411965 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:10.411940    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-fb0b93-7fb666d8f9-wr46f"]
Oct 30 22:04:10.414120 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:04:10.414093    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24066223_6a66_43ef_a85d_83183bc37fa7.slice/crio-fd252ce6030cbcce33bb5e0774214bdd6564de3a784bd5796a81524bf47d9507 WatchSource:0}: Error finding container fd252ce6030cbcce33bb5e0774214bdd6564de3a784bd5796a81524bf47d9507: Status 404 returned error can't find the container with id fd252ce6030cbcce33bb5e0774214bdd6564de3a784bd5796a81524bf47d9507
Oct 30 22:04:10.574706 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:04:10.574605    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 22:04:11.146782 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:11.146742    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-fb0b93-7fb666d8f9-wr46f" event={"ID":"24066223-6a66-43ef-a85d-83183bc37fa7","Type":"ContainerStarted","Data":"caef6a84c7871725c67e240f5e7d0a2b96ce07698edb2d00fd3566e1e8e78a98"}
Oct 30 22:04:11.147314 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:11.147290    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-fb0b93-7fb666d8f9-wr46f" event={"ID":"24066223-6a66-43ef-a85d-83183bc37fa7","Type":"ContainerStarted","Data":"fd252ce6030cbcce33bb5e0774214bdd6564de3a784bd5796a81524bf47d9507"}
Oct 30 22:04:11.147488 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:11.147470    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-fb0b93-7fb666d8f9-wr46f"
Oct 30 22:04:11.261698 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:11.261643    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-fb0b93-7fb666d8f9-wr46f" podStartSLOduration=2.261626714 podStartE2EDuration="2.261626714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:11.257904286 +0000 UTC m=+828.862211841" watchObservedRunningTime="2026-10-30 22:04:11.261626714 +0000 UTC m=+828.865934266"
Oct 30 22:04:12.073999 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:12.073632    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-bc714d-57765d6599-txzbm"
Oct 30 22:04:12.074259 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:12.074240    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-bc714d-7c5b5598f-phvl6"
Oct 30 22:04:12.074348 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:12.074320    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-779f32-66c9bd7459-g2qwk"
Oct 30 22:04:12.178368 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:12.178330    5678 generic.go:356] "Generic (PLEG): container finished" podID="99f99481-73d0-4fb4-9c88-12ee143a6904" containerID="1850898ccd8dc9a1d5251132a95b3f1e312f10352d5a5baaea7fc97ccbe83434" exitCode=0
Oct 30 22:04:12.178825 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:12.178421    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-wqqzd" event={"ID":"99f99481-73d0-4fb4-9c88-12ee143a6904","Type":"ContainerDied","Data":"1850898ccd8dc9a1d5251132a95b3f1e312f10352d5a5baaea7fc97ccbe83434"}
Oct 30 22:04:12.178890 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:12.178877    5678 scope.go:122] "RemoveContainer" containerID="1850898ccd8dc9a1d5251132a95b3f1e312f10352d5a5baaea7fc97ccbe83434"
Oct 30 22:04:12.180077 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:12.180054    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:04:12.180150 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:12.180116    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:04:12.180185 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:12.180151    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:04:13.206435 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:13.205471    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-wqqzd" event={"ID":"99f99481-73d0-4fb4-9c88-12ee143a6904","Type":"ContainerStarted","Data":"ba02e73767820b8f4d9f9f6c922d0537157b67d8705891fb420aabb534e608c4"}
Oct 30 22:04:13.285426 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:13.282639    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-wqqzd"
Oct 30 22:04:13.285426 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:13.284042    5678 patch_prober.go:29] interesting pod/e2e-worker-w2-bc714d-openshift-default-67558b95cf-wqqzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Oct 30 22:04:13.285426 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:13.284093    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-wqqzd" podUID="99f99481-73d0-4fb4-9c88-12ee143a6904" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Oct 30 22:04:14.095425 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:14.095071    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a00d9b-55d8cbb68c-dvsfc"
Oct 30 22:04:14.270428 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:14.269797    5678 patch_prober.go:29] interesting pod/e2e-worker-w2-bc714d-openshift-default-67558b95cf-wqqzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Oct 30 22:04:14.270428 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:14.269860    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-wqqzd" podUID="99f99481-73d0-4fb4-9c88-12ee143a6904" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Oct 30 22:04:14.954424 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:14.953485    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-wqqzd"
Oct 30 22:04:15.272484 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:15.270792    5678 patch_prober.go:29] interesting pod/e2e-worker-w2-bc714d-openshift-default-67558b95cf-wqqzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Oct 30 22:04:15.272484 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:15.270854    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-wqqzd" podUID="99f99481-73d0-4fb4-9c88-12ee143a6904" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Oct 30 22:04:16.271217 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:16.271178    5678 patch_prober.go:29] interesting pod/e2e-worker-w2-bc714d-openshift-default-67558b95cf-wqqzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Oct 30 22:04:16.271569 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:16.271541    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-wqqzd" podUID="99f99481-73d0-4fb4-9c88-12ee143a6904" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Oct 30 22:04:17.270558 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:17.270481    5678 patch_prober.go:29] interesting pod/e2e-worker-w2-bc714d-openshift-default-67558b95cf-wqqzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Oct 30 22:04:17.270558 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:17.270542    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-wqqzd" podUID="99f99481-73d0-4fb4-9c88-12ee143a6904" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Oct 30 22:04:18.269930 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:18.269887    5678 patch_prober.go:29] interesting pod/e2e-worker-w2-bc714d-openshift-default-67558b95cf-wqqzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Oct 30 22:04:18.270153 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:18.269952    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-wqqzd" podUID="99f99481-73d0-4fb4-9c88-12ee143a6904" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Oct 30 22:04:19.274092 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:19.274061    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-wqqzd"
Oct 30 22:04:19.275439 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:19.275395    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-wqqzd"
Oct 30 22:04:20.136826 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:20.135804    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-65e59f-74d8d9b4c-t64kq"
Oct 30 22:04:21.282574 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:21.282497    5678 generic.go:356] "Generic (PLEG): container finished" podID="4618a35f-6775-4be2-ab02-e5c9940e13b0" containerID="656ee0b1af412dbc9cc40e18f123f9e2dc69a6ac2770b6536e90afe20e196612" exitCode=0
Oct 30 22:04:21.284285 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:21.282580    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cdbcf" event={"ID":"4618a35f-6775-4be2-ab02-e5c9940e13b0","Type":"ContainerDied","Data":"656ee0b1af412dbc9cc40e18f123f9e2dc69a6ac2770b6536e90afe20e196612"}
Oct 30 22:04:21.284285 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:21.283042    5678 scope.go:122] "RemoveContainer" containerID="656ee0b1af412dbc9cc40e18f123f9e2dc69a6ac2770b6536e90afe20e196612"
Oct 30 22:04:21.284978 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:21.284505    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:04:21.284978 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:21.284605    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:04:21.284978 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:21.284651    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:04:22.289368 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:22.289331    5678 generic.go:356] "Generic (PLEG): container finished" podID="e400e450-98f2-423b-bbc1-478fb88f61a2" containerID="6b480b2933e1c4531fb1444aed1d873b1593d1a10905c1b7420b8291ce7ada3c" exitCode=0
Oct 30 22:04:22.289776 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:22.289447    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-qp9q4" event={"ID":"e400e450-98f2-423b-bbc1-478fb88f61a2","Type":"ContainerDied","Data":"6b480b2933e1c4531fb1444aed1d873b1593d1a10905c1b7420b8291ce7ada3c"}
Oct 30 22:04:22.289819 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:22.289786    5678 scope.go:122] "RemoveContainer" containerID="6b480b2933e1c4531fb1444aed1d873b1593d1a10905c1b7420b8291ce7ada3c"
Oct 30 22:04:22.294417 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:22.291268    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:04:22.294417 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:22.291357    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:04:22.294417 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:22.291456    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:04:22.296628 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:22.296545    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cdbcf" event={"ID":"4618a35f-6775-4be2-ab02-e5c9940e13b0","Type":"ContainerStarted","Data":"0f982004d8444fe33ae8a1407a7f808eabc15c117e0d2bacfa68885ce6d11d97"}
Oct 30 22:04:22.591953 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:22.591823    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cdbcf"
Oct 30 22:04:22.592508 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:22.592200    5678 patch_prober.go:29] interesting pod/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cdbcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Oct 30 22:04:22.592508 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:22.592263    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cdbcf" podUID="4618a35f-6775-4be2-ab02-e5c9940e13b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Oct 30 22:04:23.208590 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:23.208556    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-fb0b93-7fb666d8f9-wr46f"
Oct 30 22:04:23.308491 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:23.308453    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-qp9q4" event={"ID":"e400e450-98f2-423b-bbc1-478fb88f61a2","Type":"ContainerStarted","Data":"fdeb0d71791c9066e532453a2f5662b6bd9cea6cdfea15e7c29496a2e09ca469"}
Oct 30 22:04:23.592913 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:23.592796    5678 patch_prober.go:29] interesting pod/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cdbcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Oct 30 22:04:23.593186 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:23.593144    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cdbcf" podUID="4618a35f-6775-4be2-ab02-e5c9940e13b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Oct 30 22:04:23.734787 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:23.734754    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-qp9q4"
Oct 30 22:04:23.735157 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:23.735050    5678 patch_prober.go:29] interesting pod/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-qp9q4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Oct 30 22:04:23.735320 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:23.735297    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-qp9q4" podUID="e400e450-98f2-423b-bbc1-478fb88f61a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Oct 30 22:04:24.595807 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:24.595771    5678 patch_prober.go:29] interesting pod/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cdbcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Oct 30 22:04:24.596428 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:24.596375    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cdbcf" podUID="4618a35f-6775-4be2-ab02-e5c9940e13b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Oct 30 22:04:24.739426 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:24.738815    5678 patch_prober.go:29] interesting pod/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-qp9q4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Oct 30 22:04:24.739426 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:24.738879    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-qp9q4" podUID="e400e450-98f2-423b-bbc1-478fb88f61a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Oct 30 22:04:25.591692 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:25.591663    5678 patch_prober.go:29] interesting pod/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cdbcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Oct 30 22:04:25.591911 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:25.591889    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cdbcf" podUID="4618a35f-6775-4be2-ab02-e5c9940e13b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Oct 30 22:04:25.735245 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:25.734757    5678 patch_prober.go:29] interesting pod/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-qp9q4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Oct 30 22:04:25.735245 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:25.734819    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-qp9q4" podUID="e400e450-98f2-423b-bbc1-478fb88f61a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Oct 30 22:04:26.139425 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:26.133676    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-65e59f-555d4ff49c-f8ncb"]
Oct 30 22:04:26.153183 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:26.153156    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-65e59f-555d4ff49c-f8ncb"
Oct 30 22:04:26.190506 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:26.190431    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-65e59f-555d4ff49c-f8ncb"]
Oct 30 22:04:26.257426 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:26.233829    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-jglqk\""
Oct 30 22:04:26.257426 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:26.233942    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:04:26.257426 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:26.254745    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 30 22:04:26.257426 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:26.254842    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:04:26.257426 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:26.254872    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-65e59f-serving-cert\""
Oct 30 22:04:26.278437 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:26.277018    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2834e126-aa61-40fd-86fd-a548211df309-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-65e59f-555d4ff49c-f8ncb\" (UID: \"2834e126-aa61-40fd-86fd-a548211df309\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-65e59f-555d4ff49c-f8ncb"
Oct 30 22:04:26.278437 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:26.277175    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2834e126-aa61-40fd-86fd-a548211df309-metrics-certs\") pod \"maas-api-e2e-worker-w4-65e59f-555d4ff49c-f8ncb\" (UID: \"2834e126-aa61-40fd-86fd-a548211df309\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-65e59f-555d4ff49c-f8ncb"
Oct 30 22:04:26.278437 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:26.277200    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9k765\" (UniqueName: \"kubernetes.io/projected/2834e126-aa61-40fd-86fd-a548211df309-kube-api-access-9k765\") pod \"maas-api-e2e-worker-w4-65e59f-555d4ff49c-f8ncb\" (UID: \"2834e126-aa61-40fd-86fd-a548211df309\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-65e59f-555d4ff49c-f8ncb"
Oct 30 22:04:26.278437 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:26.277316    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2834e126-aa61-40fd-86fd-a548211df309-maas-api-tls\") pod \"maas-api-e2e-worker-w4-65e59f-555d4ff49c-f8ncb\" (UID: \"2834e126-aa61-40fd-86fd-a548211df309\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-65e59f-555d4ff49c-f8ncb"
Oct 30 22:04:26.378321 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:26.378272    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2834e126-aa61-40fd-86fd-a548211df309-metrics-certs\") pod \"maas-api-e2e-worker-w4-65e59f-555d4ff49c-f8ncb\" (UID: \"2834e126-aa61-40fd-86fd-a548211df309\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-65e59f-555d4ff49c-f8ncb"
Oct 30 22:04:26.378321 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:26.378331    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9k765\" (UniqueName: \"kubernetes.io/projected/2834e126-aa61-40fd-86fd-a548211df309-kube-api-access-9k765\") pod \"maas-api-e2e-worker-w4-65e59f-555d4ff49c-f8ncb\" (UID: \"2834e126-aa61-40fd-86fd-a548211df309\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-65e59f-555d4ff49c-f8ncb"
Oct 30 22:04:26.378670 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:26.378431    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2834e126-aa61-40fd-86fd-a548211df309-maas-api-tls\") pod \"maas-api-e2e-worker-w4-65e59f-555d4ff49c-f8ncb\" (UID: \"2834e126-aa61-40fd-86fd-a548211df309\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-65e59f-555d4ff49c-f8ncb"
Oct 30 22:04:26.378670 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:26.378517    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2834e126-aa61-40fd-86fd-a548211df309-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-65e59f-555d4ff49c-f8ncb\" (UID: \"2834e126-aa61-40fd-86fd-a548211df309\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-65e59f-555d4ff49c-f8ncb"
Oct 30 22:04:26.379361 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:26.379333    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2834e126-aa61-40fd-86fd-a548211df309-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-65e59f-555d4ff49c-f8ncb\" (UID: \"2834e126-aa61-40fd-86fd-a548211df309\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-65e59f-555d4ff49c-f8ncb"
Oct 30 22:04:26.383590 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:26.383563    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2834e126-aa61-40fd-86fd-a548211df309-maas-api-tls\") pod \"maas-api-e2e-worker-w4-65e59f-555d4ff49c-f8ncb\" (UID: \"2834e126-aa61-40fd-86fd-a548211df309\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-65e59f-555d4ff49c-f8ncb"
Oct 30 22:04:26.384059 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:26.384010    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2834e126-aa61-40fd-86fd-a548211df309-metrics-certs\") pod \"maas-api-e2e-worker-w4-65e59f-555d4ff49c-f8ncb\" (UID: \"2834e126-aa61-40fd-86fd-a548211df309\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-65e59f-555d4ff49c-f8ncb"
Oct 30 22:04:26.463477 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:26.463083    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9k765\" (UniqueName: \"kubernetes.io/projected/2834e126-aa61-40fd-86fd-a548211df309-kube-api-access-9k765\") pod \"maas-api-e2e-worker-w4-65e59f-555d4ff49c-f8ncb\" (UID: \"2834e126-aa61-40fd-86fd-a548211df309\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-65e59f-555d4ff49c-f8ncb"
Oct 30 22:04:26.479065 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:26.478439    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-65e59f-555d4ff49c-f8ncb"
Oct 30 22:04:26.591879 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:26.591841    5678 patch_prober.go:29] interesting pod/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cdbcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Oct 30 22:04:26.592038 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:26.591911    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cdbcf" podUID="4618a35f-6775-4be2-ab02-e5c9940e13b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Oct 30 22:04:26.743475 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:26.734766    5678 patch_prober.go:29] interesting pod/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-qp9q4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Oct 30 22:04:26.743475 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:26.734823    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-qp9q4" podUID="e400e450-98f2-423b-bbc1-478fb88f61a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Oct 30 22:04:26.970461 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:26.964121    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-65e59f-555d4ff49c-f8ncb"]
Oct 30 22:04:26.972567 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:04:26.971137    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2834e126_aa61_40fd_86fd_a548211df309.slice/crio-c810eb736a9c51f6c01b00c9a0bf4f544bad66215bf2482006200af189f3c84b WatchSource:0}: Error finding container c810eb736a9c51f6c01b00c9a0bf4f544bad66215bf2482006200af189f3c84b: Status 404 returned error can't find the container with id c810eb736a9c51f6c01b00c9a0bf4f544bad66215bf2482006200af189f3c84b
Oct 30 22:04:27.392643 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:27.392602    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-65e59f-555d4ff49c-f8ncb" event={"ID":"2834e126-aa61-40fd-86fd-a548211df309","Type":"ContainerStarted","Data":"c810eb736a9c51f6c01b00c9a0bf4f544bad66215bf2482006200af189f3c84b"}
Oct 30 22:04:27.639806 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:27.639761    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cdbcf"
Oct 30 22:04:27.641745 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:27.641724    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cdbcf"
Oct 30 22:04:27.647870 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:27.647806    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cdbcf"
Oct 30 22:04:27.737196 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:27.737160    5678 patch_prober.go:29] interesting pod/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-qp9q4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Oct 30 22:04:27.737462 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:27.737434    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-qp9q4" podUID="e400e450-98f2-423b-bbc1-478fb88f61a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Oct 30 22:04:28.838492 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:28.837807    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-qp9q4"
Oct 30 22:04:28.838901 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:28.838619    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-qp9q4"
Oct 30 22:04:28.839880 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:28.839853    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-qp9q4"
Oct 30 22:04:29.206571 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:29.206537    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-6bjjs"]
Oct 30 22:04:29.209727 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:29.209704    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-6bjjs"
Oct 30 22:04:29.308589 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:29.308559    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-6bjjs"]
Oct 30 22:04:29.314808 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:29.314779    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb-istio-data\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-6bjjs\" (UID: \"275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-6bjjs"
Oct 30 22:04:29.314960 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:29.314820    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb-workload-socket\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-6bjjs\" (UID: \"275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-6bjjs"
Oct 30 22:04:29.314960 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:29.314886    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb-istio-envoy\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-6bjjs\" (UID: \"275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-6bjjs"
Oct 30 22:04:29.314960 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:29.314936    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb-istio-token\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-6bjjs\" (UID: \"275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-6bjjs"
Oct 30 22:04:29.315073 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:29.314967    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb-istio-podinfo\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-6bjjs\" (UID: \"275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-6bjjs"
Oct 30 22:04:29.315073 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:29.315000    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb-credential-socket\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-6bjjs\" (UID: \"275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-6bjjs"
Oct 30 22:04:29.315073 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:29.315022    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb-istiod-ca-cert\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-6bjjs\" (UID: \"275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-6bjjs"
Oct 30 22:04:29.315073 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:29.315037    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76nnv\" (UniqueName: \"kubernetes.io/projected/275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb-kube-api-access-76nnv\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-6bjjs\" (UID: \"275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-6bjjs"
Oct 30 22:04:29.315073 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:29.315055    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb-workload-certs\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-6bjjs\" (UID: \"275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-6bjjs"
Oct 30 22:04:29.415677 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:29.415646    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb-istio-data\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-6bjjs\" (UID: \"275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-6bjjs"
Oct 30 22:04:29.415828 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:29.415699    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb-workload-socket\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-6bjjs\" (UID: \"275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-6bjjs"
Oct 30 22:04:29.415902 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:29.415836    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb-istio-envoy\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-6bjjs\" (UID: \"275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-6bjjs"
Oct 30 22:04:29.415959 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:29.415902    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb-istio-token\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-6bjjs\" (UID: \"275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-6bjjs"
Oct 30 22:04:29.416019 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:29.415962    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb-istio-podinfo\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-6bjjs\" (UID: \"275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-6bjjs"
Oct 30 22:04:29.416075 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:29.416025    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb-credential-socket\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-6bjjs\" (UID: \"275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-6bjjs"
Oct 30 22:04:29.416075 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:29.416063    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb-istiod-ca-cert\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-6bjjs\" (UID: \"275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-6bjjs"
Oct 30 22:04:29.416186 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:29.416089    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76nnv\" (UniqueName: \"kubernetes.io/projected/275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb-kube-api-access-76nnv\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-6bjjs\" (UID: \"275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-6bjjs"
Oct 30 22:04:29.416186 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:29.416123    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb-workload-certs\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-6bjjs\" (UID: \"275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-6bjjs"
Oct 30 22:04:29.416186 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:29.416136    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb-istio-data\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-6bjjs\" (UID: \"275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-6bjjs"
Oct 30 22:04:29.416186 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:29.416123    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb-workload-socket\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-6bjjs\" (UID: \"275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-6bjjs"
Oct 30 22:04:29.416420 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:29.416360    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb-credential-socket\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-6bjjs\" (UID: \"275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-6bjjs"
Oct 30 22:04:29.416630 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:29.416608    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb-workload-certs\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-6bjjs\" (UID: \"275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-6bjjs"
Oct 30 22:04:29.416926 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:29.416901    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb-istiod-ca-cert\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-6bjjs\" (UID: \"275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-6bjjs"
Oct 30 22:04:29.418619 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:29.418595    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb-istio-envoy\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-6bjjs\" (UID: \"275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-6bjjs"
Oct 30 22:04:29.418775 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:29.418752    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb-istio-podinfo\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-6bjjs\" (UID: \"275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-6bjjs"
Oct 30 22:04:29.474566 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:29.474534    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb-istio-token\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-6bjjs\" (UID: \"275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-6bjjs"
Oct 30 22:04:29.474677 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:29.474579    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76nnv\" (UniqueName: \"kubernetes.io/projected/275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb-kube-api-access-76nnv\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-6bjjs\" (UID: \"275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-6bjjs"
Oct 30 22:04:29.524824 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:29.524799    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-6bjjs"
Oct 30 22:04:29.959391 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:29.959364    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-6bjjs"]
Oct 30 22:04:29.961250 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:04:29.961221    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod275c9ec2_0c92_4e50_b6b7_a1f82c02d3eb.slice/crio-c132fa00e8c358c42c15e7c8266725394f4b9e86c6ed3f5c185317865911e6b8 WatchSource:0}: Error finding container c132fa00e8c358c42c15e7c8266725394f4b9e86c6ed3f5c185317865911e6b8: Status 404 returned error can't find the container with id c132fa00e8c358c42c15e7c8266725394f4b9e86c6ed3f5c185317865911e6b8
Oct 30 22:04:29.963624 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:29.963592    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:04:29.963701 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:29.963660    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:04:29.963701 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:29.963689    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:04:30.429956 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:30.429901    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-65e59f-555d4ff49c-f8ncb" event={"ID":"2834e126-aa61-40fd-86fd-a548211df309","Type":"ContainerStarted","Data":"59d1ff72a3743020c6e593e5c8addad669867d00a51cdfc307dda3a0925db52b"}
Oct 30 22:04:30.430687 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:30.430658    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-65e59f-555d4ff49c-f8ncb"
Oct 30 22:04:30.433689 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:30.433659    5678 generic.go:356] "Generic (PLEG): container finished" podID="dbbbcda0-3462-4c73-af87-c1c8fcca4706" containerID="495ad04d6a89274ae44c70c17d1fc9defd3d924808919743235cb5ea283530fb" exitCode=0
Oct 30 22:04:30.434024 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:30.434002    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9fln8" event={"ID":"dbbbcda0-3462-4c73-af87-c1c8fcca4706","Type":"ContainerDied","Data":"495ad04d6a89274ae44c70c17d1fc9defd3d924808919743235cb5ea283530fb"}
Oct 30 22:04:30.434668 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:30.434652    5678 scope.go:122] "RemoveContainer" containerID="495ad04d6a89274ae44c70c17d1fc9defd3d924808919743235cb5ea283530fb"
Oct 30 22:04:30.437516 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:30.437482    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:04:30.437723 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:30.437682    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:04:30.437871 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:30.437853    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:04:30.449257 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:30.449227    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-6bjjs" event={"ID":"275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb","Type":"ContainerStarted","Data":"275225ea6d2dcfed45a1f3ba0e0b1eb6dc329fd98ea33a7806a222b59d9aa280"}
Oct 30 22:04:30.449420 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:30.449284    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-6bjjs" event={"ID":"275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb","Type":"ContainerStarted","Data":"c132fa00e8c358c42c15e7c8266725394f4b9e86c6ed3f5c185317865911e6b8"}
Oct 30 22:04:30.528606 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:30.528522    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-6bjjs"
Oct 30 22:04:30.551636 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:30.551569    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-65e59f-555d4ff49c-f8ncb" podStartSLOduration=2.141877287 podStartE2EDuration="4.551548584s" totalImagesPullingTime="2.409671297s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:30.548890311 +0000 UTC m=+848.153197865" watchObservedRunningTime="2026-10-30 22:04:30.551548584 +0000 UTC m=+848.155856139"
Oct 30 22:04:30.783469 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:30.783346    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-6bjjs" podStartSLOduration=1.783310333 podStartE2EDuration="1.783310333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:30.78284113 +0000 UTC m=+848.387148933" watchObservedRunningTime="2026-10-30 22:04:30.783310333 +0000 UTC m=+848.387617888"
Oct 30 22:04:31.488261 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:31.488197    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9fln8" event={"ID":"dbbbcda0-3462-4c73-af87-c1c8fcca4706","Type":"ContainerStarted","Data":"ad3f73b59bc268bcada7aa7e7d195c6fb763b274faaa710c01d4bcffa3852d6e"}
Oct 30 22:04:31.556555 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:31.556519    5678 patch_prober.go:29] interesting pod/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-6bjjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Oct 30 22:04:31.556814 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:31.556790    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-6bjjs" podUID="275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Oct 30 22:04:31.978896 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:31.978854    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9fln8"
Oct 30 22:04:32.004602 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:32.004575    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9fln8"
Oct 30 22:04:32.006024 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:32.005994    5678 patch_prober.go:29] interesting pod/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9fln8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Oct 30 22:04:32.006146 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:32.006044    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9fln8" podUID="dbbbcda0-3462-4c73-af87-c1c8fcca4706" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Oct 30 22:04:32.542853 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:32.542821    5678 patch_prober.go:29] interesting pod/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-6bjjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Oct 30 22:04:32.543446 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:32.543416    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-6bjjs" podUID="275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Oct 30 22:04:33.005828 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:33.005798    5678 patch_prober.go:29] interesting pod/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9fln8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Oct 30 22:04:33.006061 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:33.006042    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9fln8" podUID="dbbbcda0-3462-4c73-af87-c1c8fcca4706" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Oct 30 22:04:33.529164 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:33.528731    5678 patch_prober.go:29] interesting pod/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-6bjjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Oct 30 22:04:33.529164 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:33.528790    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-6bjjs" podUID="275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Oct 30 22:04:34.005242 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:34.005206    5678 patch_prober.go:29] interesting pod/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9fln8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Oct 30 22:04:34.005677 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:34.005255    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9fln8" podUID="dbbbcda0-3462-4c73-af87-c1c8fcca4706" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Oct 30 22:04:34.526152 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:34.525730    5678 patch_prober.go:29] interesting pod/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-6bjjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Oct 30 22:04:34.526152 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:34.525785    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-6bjjs" podUID="275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Oct 30 22:04:35.005797 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:35.005764    5678 patch_prober.go:29] interesting pod/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9fln8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Oct 30 22:04:35.006221 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:35.005815    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9fln8" podUID="dbbbcda0-3462-4c73-af87-c1c8fcca4706" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Oct 30 22:04:35.525810 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:35.525759    5678 patch_prober.go:29] interesting pod/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-6bjjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Oct 30 22:04:35.526111 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:35.526078    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-6bjjs" podUID="275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Oct 30 22:04:36.005372 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:36.005339    5678 patch_prober.go:29] interesting pod/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9fln8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Oct 30 22:04:36.005569 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:36.005436    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9fln8" podUID="dbbbcda0-3462-4c73-af87-c1c8fcca4706" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Oct 30 22:04:36.495300 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:36.495267    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-65e59f-555d4ff49c-f8ncb"
Oct 30 22:04:36.530057 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:36.530026    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-6bjjs"
Oct 30 22:04:36.530433 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:36.530416    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-6bjjs"
Oct 30 22:04:36.531157 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:36.531130    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-6bjjs"
Oct 30 22:04:37.009736 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:37.009707    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9fln8"
Oct 30 22:04:37.010809 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:04:37.010791    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9fln8"
Oct 30 22:04:40.572120 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:04:40.572085    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 22:05:10.571962 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:05:10.571918    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 22:05:19.977253 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:05:19.977222    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-fb0b93/simulator-w6-fb0b93-kserve-55b784b479-588bn"]
Oct 30 22:05:19.980437 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:05:19.980418    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-fb0b93/simulator-w6-fb0b93-kserve-55b784b479-588bn"
Oct 30 22:05:20.033740 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:05:20.033708    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/951a6f7e-a22e-40f2-ace2-1ce5d3bc9b49-tmp-dir\") pod \"simulator-w6-fb0b93-kserve-55b784b479-588bn\" (UID: \"951a6f7e-a22e-40f2-ace2-1ce5d3bc9b49\") " pod="e2e-models-e2e-worker-w6-fb0b93/simulator-w6-fb0b93-kserve-55b784b479-588bn"
Oct 30 22:05:20.033909 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:05:20.033750    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/951a6f7e-a22e-40f2-ace2-1ce5d3bc9b49-home\") pod \"simulator-w6-fb0b93-kserve-55b784b479-588bn\" (UID: \"951a6f7e-a22e-40f2-ace2-1ce5d3bc9b49\") " pod="e2e-models-e2e-worker-w6-fb0b93/simulator-w6-fb0b93-kserve-55b784b479-588bn"
Oct 30 22:05:20.033909 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:05:20.033881    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/951a6f7e-a22e-40f2-ace2-1ce5d3bc9b49-dshm\") pod \"simulator-w6-fb0b93-kserve-55b784b479-588bn\" (UID: \"951a6f7e-a22e-40f2-ace2-1ce5d3bc9b49\") " pod="e2e-models-e2e-worker-w6-fb0b93/simulator-w6-fb0b93-kserve-55b784b479-588bn"
Oct 30 22:05:20.033982 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:05:20.033961    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/951a6f7e-a22e-40f2-ace2-1ce5d3bc9b49-model-cache\") pod \"simulator-w6-fb0b93-kserve-55b784b479-588bn\" (UID: \"951a6f7e-a22e-40f2-ace2-1ce5d3bc9b49\") " pod="e2e-models-e2e-worker-w6-fb0b93/simulator-w6-fb0b93-kserve-55b784b479-588bn"
Oct 30 22:05:20.034040 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:05:20.034025    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmpvb\" (UniqueName: \"kubernetes.io/projected/951a6f7e-a22e-40f2-ace2-1ce5d3bc9b49-kube-api-access-jmpvb\") pod \"simulator-w6-fb0b93-kserve-55b784b479-588bn\" (UID: \"951a6f7e-a22e-40f2-ace2-1ce5d3bc9b49\") " pod="e2e-models-e2e-worker-w6-fb0b93/simulator-w6-fb0b93-kserve-55b784b479-588bn"
Oct 30 22:05:20.034132 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:05:20.034120    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/951a6f7e-a22e-40f2-ace2-1ce5d3bc9b49-tls-certs\") pod \"simulator-w6-fb0b93-kserve-55b784b479-588bn\" (UID: \"951a6f7e-a22e-40f2-ace2-1ce5d3bc9b49\") " pod="e2e-models-e2e-worker-w6-fb0b93/simulator-w6-fb0b93-kserve-55b784b479-588bn"
Oct 30 22:05:20.044452 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:05:20.044423    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-fb0b93\"/\"openshift-service-ca.crt\""
Oct 30 22:05:20.077503 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:05:20.077478    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-fb0b93\"/\"default-dockercfg-vssbl\""
Oct 30 22:05:20.077641 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:05:20.077528    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-fb0b93\"/\"kube-root-ca.crt\""
Oct 30 22:05:20.077641 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:05:20.077629    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-fb0b93\"/\"simulator-w6-fb0b93-kserve-self-signed-certs\""
Oct 30 22:05:20.080382 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:05:20.080361    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-fb0b93/simulator-w6-fb0b93-kserve-55b784b479-588bn"]
Oct 30 22:05:20.135523 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:05:20.135498    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/951a6f7e-a22e-40f2-ace2-1ce5d3bc9b49-dshm\") pod \"simulator-w6-fb0b93-kserve-55b784b479-588bn\" (UID: \"951a6f7e-a22e-40f2-ace2-1ce5d3bc9b49\") " pod="e2e-models-e2e-worker-w6-fb0b93/simulator-w6-fb0b93-kserve-55b784b479-588bn"
Oct 30 22:05:20.135657 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:05:20.135549    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/951a6f7e-a22e-40f2-ace2-1ce5d3bc9b49-model-cache\") pod \"simulator-w6-fb0b93-kserve-55b784b479-588bn\" (UID: \"951a6f7e-a22e-40f2-ace2-1ce5d3bc9b49\") " pod="e2e-models-e2e-worker-w6-fb0b93/simulator-w6-fb0b93-kserve-55b784b479-588bn"
Oct 30 22:05:20.135657 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:05:20.135581    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmpvb\" (UniqueName: \"kubernetes.io/projected/951a6f7e-a22e-40f2-ace2-1ce5d3bc9b49-kube-api-access-jmpvb\") pod \"simulator-w6-fb0b93-kserve-55b784b479-588bn\" (UID: \"951a6f7e-a22e-40f2-ace2-1ce5d3bc9b49\") " pod="e2e-models-e2e-worker-w6-fb0b93/simulator-w6-fb0b93-kserve-55b784b479-588bn"
Oct 30 22:05:20.135657 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:05:20.135645    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/951a6f7e-a22e-40f2-ace2-1ce5d3bc9b49-tls-certs\") pod \"simulator-w6-fb0b93-kserve-55b784b479-588bn\" (UID: \"951a6f7e-a22e-40f2-ace2-1ce5d3bc9b49\") " pod="e2e-models-e2e-worker-w6-fb0b93/simulator-w6-fb0b93-kserve-55b784b479-588bn"
Oct 30 22:05:20.135785 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:05:20.135663    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/951a6f7e-a22e-40f2-ace2-1ce5d3bc9b49-tmp-dir\") pod \"simulator-w6-fb0b93-kserve-55b784b479-588bn\" (UID: \"951a6f7e-a22e-40f2-ace2-1ce5d3bc9b49\") " pod="e2e-models-e2e-worker-w6-fb0b93/simulator-w6-fb0b93-kserve-55b784b479-588bn"
Oct 30 22:05:20.135785 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:05:20.135686    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/951a6f7e-a22e-40f2-ace2-1ce5d3bc9b49-home\") pod \"simulator-w6-fb0b93-kserve-55b784b479-588bn\" (UID: \"951a6f7e-a22e-40f2-ace2-1ce5d3bc9b49\") " pod="e2e-models-e2e-worker-w6-fb0b93/simulator-w6-fb0b93-kserve-55b784b479-588bn"
Oct 30 22:05:20.136013 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:05:20.135984    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/951a6f7e-a22e-40f2-ace2-1ce5d3bc9b49-model-cache\") pod \"simulator-w6-fb0b93-kserve-55b784b479-588bn\" (UID: \"951a6f7e-a22e-40f2-ace2-1ce5d3bc9b49\") " pod="e2e-models-e2e-worker-w6-fb0b93/simulator-w6-fb0b93-kserve-55b784b479-588bn"
Oct 30 22:05:20.136013 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:05:20.136007    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/951a6f7e-a22e-40f2-ace2-1ce5d3bc9b49-tmp-dir\") pod \"simulator-w6-fb0b93-kserve-55b784b479-588bn\" (UID: \"951a6f7e-a22e-40f2-ace2-1ce5d3bc9b49\") " pod="e2e-models-e2e-worker-w6-fb0b93/simulator-w6-fb0b93-kserve-55b784b479-588bn"
Oct 30 22:05:20.136145 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:05:20.136053    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/951a6f7e-a22e-40f2-ace2-1ce5d3bc9b49-home\") pod \"simulator-w6-fb0b93-kserve-55b784b479-588bn\" (UID: \"951a6f7e-a22e-40f2-ace2-1ce5d3bc9b49\") " pod="e2e-models-e2e-worker-w6-fb0b93/simulator-w6-fb0b93-kserve-55b784b479-588bn"
Oct 30 22:05:20.137740 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:05:20.137715    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/951a6f7e-a22e-40f2-ace2-1ce5d3bc9b49-dshm\") pod \"simulator-w6-fb0b93-kserve-55b784b479-588bn\" (UID: \"951a6f7e-a22e-40f2-ace2-1ce5d3bc9b49\") " pod="e2e-models-e2e-worker-w6-fb0b93/simulator-w6-fb0b93-kserve-55b784b479-588bn"
Oct 30 22:05:20.138004 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:05:20.137985    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/951a6f7e-a22e-40f2-ace2-1ce5d3bc9b49-tls-certs\") pod \"simulator-w6-fb0b93-kserve-55b784b479-588bn\" (UID: \"951a6f7e-a22e-40f2-ace2-1ce5d3bc9b49\") " pod="e2e-models-e2e-worker-w6-fb0b93/simulator-w6-fb0b93-kserve-55b784b479-588bn"
Oct 30 22:05:20.194158 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:05:20.194128    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmpvb\" (UniqueName: \"kubernetes.io/projected/951a6f7e-a22e-40f2-ace2-1ce5d3bc9b49-kube-api-access-jmpvb\") pod \"simulator-w6-fb0b93-kserve-55b784b479-588bn\" (UID: \"951a6f7e-a22e-40f2-ace2-1ce5d3bc9b49\") " pod="e2e-models-e2e-worker-w6-fb0b93/simulator-w6-fb0b93-kserve-55b784b479-588bn"
Oct 30 22:05:20.291821 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:05:20.291717    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-fb0b93/simulator-w6-fb0b93-kserve-55b784b479-588bn"
Oct 30 22:05:20.670269 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:05:20.670246    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-fb0b93/simulator-w6-fb0b93-kserve-55b784b479-588bn"]
Oct 30 22:05:20.672350 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:05:20.672321    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod951a6f7e_a22e_40f2_ace2_1ce5d3bc9b49.slice/crio-ff061666f7b2e8f3da57580f94b59ba143e2cd93523abc06fadb8973aafb27dc WatchSource:0}: Error finding container ff061666f7b2e8f3da57580f94b59ba143e2cd93523abc06fadb8973aafb27dc: Status 404 returned error can't find the container with id ff061666f7b2e8f3da57580f94b59ba143e2cd93523abc06fadb8973aafb27dc
Oct 30 22:05:20.763010 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:05:20.762974    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-fb0b93/simulator-w6-fb0b93-kserve-55b784b479-588bn" event={"ID":"951a6f7e-a22e-40f2-ace2-1ce5d3bc9b49","Type":"ContainerStarted","Data":"e5809108b6519b3887dd567430d21c8ad497ea330ec6fd117ba229c8435fbda5"}
Oct 30 22:05:20.763154 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:05:20.763022    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-fb0b93/simulator-w6-fb0b93-kserve-55b784b479-588bn" event={"ID":"951a6f7e-a22e-40f2-ace2-1ce5d3bc9b49","Type":"ContainerStarted","Data":"ff061666f7b2e8f3da57580f94b59ba143e2cd93523abc06fadb8973aafb27dc"}
Oct 30 22:05:20.894692 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:05:20.894642    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-fb0b93/simulator-w6-fb0b93-kserve-55b784b479-588bn" podStartSLOduration=1.894623427 podStartE2EDuration="1.894623427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:05:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:05:20.893584669 +0000 UTC m=+898.497892225" watchObservedRunningTime="2026-10-30 22:05:20.894623427 +0000 UTC m=+898.498930981"
Oct 30 22:05:21.292572 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:05:21.292542    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-fb0b93/simulator-w6-fb0b93-kserve-55b784b479-588bn"
Oct 30 22:05:30.292533 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:05:30.292493    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-fb0b93/simulator-w6-fb0b93-kserve-55b784b479-588bn"
Oct 30 22:05:30.306326 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:05:30.306297    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-fb0b93/simulator-w6-fb0b93-kserve-55b784b479-588bn"
Oct 30 22:05:30.830839 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:05:30.830806    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-fb0b93/simulator-w6-fb0b93-kserve-55b784b479-588bn"
Oct 30 22:05:40.571207 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:05:40.571126    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 22:05:41.473151 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:05:41.473113    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-7bb1c2/simulator-w3-7bb1c2-kserve-675645869c-vcwqv"]
Oct 30 22:05:41.476100 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:05:41.476077    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-7bb1c2/simulator-w3-7bb1c2-kserve-675645869c-vcwqv"
Oct 30 22:05:41.558199 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:05:41.558166    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1fce9daf-68c9-41d0-825b-2012bb93c8ba-home\") pod \"simulator-w3-7bb1c2-kserve-675645869c-vcwqv\" (UID: \"1fce9daf-68c9-41d0-825b-2012bb93c8ba\") " pod="e2e-models-e2e-worker-w3-7bb1c2/simulator-w3-7bb1c2-kserve-675645869c-vcwqv"
Oct 30 22:05:41.558354 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:05:41.558293    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fce9daf-68c9-41d0-825b-2012bb93c8ba-dshm\") pod \"simulator-w3-7bb1c2-kserve-675645869c-vcwqv\" (UID: \"1fce9daf-68c9-41d0-825b-2012bb93c8ba\") " pod="e2e-models-e2e-worker-w3-7bb1c2/simulator-w3-7bb1c2-kserve-675645869c-vcwqv"
Oct 30 22:05:41.558354 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:05:41.558334    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fce9daf-68c9-41d0-825b-2012bb93c8ba-tls-certs\") pod \"simulator-w3-7bb1c2-kserve-675645869c-vcwqv\" (UID: \"1fce9daf-68c9-41d0-825b-2012bb93c8ba\") " pod="e2e-models-e2e-worker-w3-7bb1c2/simulator-w3-7bb1c2-kserve-675645869c-vcwqv"
Oct 30 22:05:41.558467 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:05:41.558365    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fce9daf-68c9-41d0-825b-2012bb93c8ba-model-cache\") pod \"simulator-w3-7bb1c2-kserve-675645869c-vcwqv\" (UID: \"1fce9daf-68c9-41d0-825b-2012bb93c8ba\") " pod="e2e-models-e2e-worker-w3-7bb1c2/simulator-w3-7bb1c2-kserve-675645869c-vcwqv"
Oct 30 22:05:41.558542 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:05:41.558523    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tf48\" (UniqueName: \"kubernetes.io/projected/1fce9daf-68c9-41d0-825b-2012bb93c8ba-kube-api-access-8tf48\") pod \"simulator-w3-7bb1c2-kserve-675645869c-vcwqv\" (UID: \"1fce9daf-68c9-41d0-825b-2012bb93c8ba\") " pod="e2e-models-e2e-worker-w3-7bb1c2/simulator-w3-7bb1c2-kserve-675645869c-vcwqv"
Oct 30 22:05:41.558596 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:05:41.558581    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fce9daf-68c9-41d0-825b-2012bb93c8ba-tmp-dir\") pod \"simulator-w3-7bb1c2-kserve-675645869c-vcwqv\" (UID: \"1fce9daf-68c9-41d0-825b-2012bb93c8ba\") " pod="e2e-models-e2e-worker-w3-7bb1c2/simulator-w3-7bb1c2-kserve-675645869c-vcwqv"
Oct 30 22:05:41.576276 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:05:41.576244    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-7bb1c2\"/\"openshift-service-ca.crt\""
Oct 30 22:05:41.576276 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:05:41.576254    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-7bb1c2\"/\"simulator-w3-7bb1c2-kserve-self-signed-certs\""
Oct 30 22:05:41.576701 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:05:41.576301    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-7bb1c2\"/\"default-dockercfg-jf9c9\""
Oct 30 22:05:41.576753 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:05:41.576727    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-7bb1c2\"/\"kube-root-ca.crt\""
Oct 30 22:05:41.579409 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:05:41.579380    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-7bb1c2/simulator-w3-7bb1c2-kserve-675645869c-vcwqv"]
Oct 30 22:05:41.659814 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:05:41.659775    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1fce9daf-68c9-41d0-825b-2012bb93c8ba-home\") pod \"simulator-w3-7bb1c2-kserve-675645869c-vcwqv\" (UID: \"1fce9daf-68c9-41d0-825b-2012bb93c8ba\") " pod="e2e-models-e2e-worker-w3-7bb1c2/simulator-w3-7bb1c2-kserve-675645869c-vcwqv"
Oct 30 22:05:41.659991 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:05:41.659858    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fce9daf-68c9-41d0-825b-2012bb93c8ba-dshm\") pod \"simulator-w3-7bb1c2-kserve-675645869c-vcwqv\" (UID: \"1fce9daf-68c9-41d0-825b-2012bb93c8ba\") " pod="e2e-models-e2e-worker-w3-7bb1c2/simulator-w3-7bb1c2-kserve-675645869c-vcwqv"
Oct 30 22:05:41.659991 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:05:41.659882    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fce9daf-68c9-41d0-825b-2012bb93c8ba-tls-certs\") pod \"simulator-w3-7bb1c2-kserve-675645869c-vcwqv\" (UID: \"1fce9daf-68c9-41d0-825b-2012bb93c8ba\") " pod="e2e-models-e2e-worker-w3-7bb1c2/simulator-w3-7bb1c2-kserve-675645869c-vcwqv"
Oct 30 22:05:41.659991 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:05:41.659900    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fce9daf-68c9-41d0-825b-2012bb93c8ba-model-cache\") pod \"simulator-w3-7bb1c2-kserve-675645869c-vcwqv\" (UID: \"1fce9daf-68c9-41d0-825b-2012bb93c8ba\") " pod="e2e-models-e2e-worker-w3-7bb1c2/simulator-w3-7bb1c2-kserve-675645869c-vcwqv"
Oct 30 22:05:41.659991 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:05:41.659937    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tf48\" (UniqueName: \"kubernetes.io/projected/1fce9daf-68c9-41d0-825b-2012bb93c8ba-kube-api-access-8tf48\") pod \"simulator-w3-7bb1c2-kserve-675645869c-vcwqv\" (UID: \"1fce9daf-68c9-41d0-825b-2012bb93c8ba\") " pod="e2e-models-e2e-worker-w3-7bb1c2/simulator-w3-7bb1c2-kserve-675645869c-vcwqv"
Oct 30 22:05:41.659991 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:05:41.659981    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fce9daf-68c9-41d0-825b-2012bb93c8ba-tmp-dir\") pod \"simulator-w3-7bb1c2-kserve-675645869c-vcwqv\" (UID: \"1fce9daf-68c9-41d0-825b-2012bb93c8ba\") " pod="e2e-models-e2e-worker-w3-7bb1c2/simulator-w3-7bb1c2-kserve-675645869c-vcwqv"
Oct 30 22:05:41.660297 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:05:41.660266    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1fce9daf-68c9-41d0-825b-2012bb93c8ba-home\") pod \"simulator-w3-7bb1c2-kserve-675645869c-vcwqv\" (UID: \"1fce9daf-68c9-41d0-825b-2012bb93c8ba\") " pod="e2e-models-e2e-worker-w3-7bb1c2/simulator-w3-7bb1c2-kserve-675645869c-vcwqv"
Oct 30 22:05:41.660297 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:05:41.660283    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fce9daf-68c9-41d0-825b-2012bb93c8ba-model-cache\") pod \"simulator-w3-7bb1c2-kserve-675645869c-vcwqv\" (UID: \"1fce9daf-68c9-41d0-825b-2012bb93c8ba\") " pod="e2e-models-e2e-worker-w3-7bb1c2/simulator-w3-7bb1c2-kserve-675645869c-vcwqv"
Oct 30 22:05:41.660511 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:05:41.660338    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fce9daf-68c9-41d0-825b-2012bb93c8ba-tmp-dir\") pod \"simulator-w3-7bb1c2-kserve-675645869c-vcwqv\" (UID: \"1fce9daf-68c9-41d0-825b-2012bb93c8ba\") " pod="e2e-models-e2e-worker-w3-7bb1c2/simulator-w3-7bb1c2-kserve-675645869c-vcwqv"
Oct 30 22:05:41.662118 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:05:41.662092    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fce9daf-68c9-41d0-825b-2012bb93c8ba-dshm\") pod \"simulator-w3-7bb1c2-kserve-675645869c-vcwqv\" (UID: \"1fce9daf-68c9-41d0-825b-2012bb93c8ba\") " pod="e2e-models-e2e-worker-w3-7bb1c2/simulator-w3-7bb1c2-kserve-675645869c-vcwqv"
Oct 30 22:05:41.662524 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:05:41.662505    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fce9daf-68c9-41d0-825b-2012bb93c8ba-tls-certs\") pod \"simulator-w3-7bb1c2-kserve-675645869c-vcwqv\" (UID: \"1fce9daf-68c9-41d0-825b-2012bb93c8ba\") " pod="e2e-models-e2e-worker-w3-7bb1c2/simulator-w3-7bb1c2-kserve-675645869c-vcwqv"
Oct 30 22:05:41.718715 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:05:41.718676    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tf48\" (UniqueName: \"kubernetes.io/projected/1fce9daf-68c9-41d0-825b-2012bb93c8ba-kube-api-access-8tf48\") pod \"simulator-w3-7bb1c2-kserve-675645869c-vcwqv\" (UID: \"1fce9daf-68c9-41d0-825b-2012bb93c8ba\") " pod="e2e-models-e2e-worker-w3-7bb1c2/simulator-w3-7bb1c2-kserve-675645869c-vcwqv"
Oct 30 22:05:41.787588 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:05:41.787503    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-7bb1c2/simulator-w3-7bb1c2-kserve-675645869c-vcwqv"
Oct 30 22:05:42.165131 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:05:42.165098    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-7bb1c2/simulator-w3-7bb1c2-kserve-675645869c-vcwqv"]
Oct 30 22:05:42.168648 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:05:42.168619    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1fce9daf_68c9_41d0_825b_2012bb93c8ba.slice/crio-9c659b5bb1063703e4eb6bb683994042dc86e9890c2e004c5dbc84e0e50313da WatchSource:0}: Error finding container 9c659b5bb1063703e4eb6bb683994042dc86e9890c2e004c5dbc84e0e50313da: Status 404 returned error can't find the container with id 9c659b5bb1063703e4eb6bb683994042dc86e9890c2e004c5dbc84e0e50313da
Oct 30 22:05:42.896296 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:05:42.896256    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-7bb1c2/simulator-w3-7bb1c2-kserve-675645869c-vcwqv" event={"ID":"1fce9daf-68c9-41d0-825b-2012bb93c8ba","Type":"ContainerStarted","Data":"864e0d7ba3a5a06cc6ef94c2d50658438a65d46b3fb3de3d21181d619a78c121"}
Oct 30 22:05:42.896296 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:05:42.896301    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-7bb1c2/simulator-w3-7bb1c2-kserve-675645869c-vcwqv" event={"ID":"1fce9daf-68c9-41d0-825b-2012bb93c8ba","Type":"ContainerStarted","Data":"9c659b5bb1063703e4eb6bb683994042dc86e9890c2e004c5dbc84e0e50313da"}
Oct 30 22:05:43.016863 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:05:43.016802    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-7bb1c2/simulator-w3-7bb1c2-kserve-675645869c-vcwqv" podStartSLOduration=2.016784512 podStartE2EDuration="2.016784512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:05:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:05:43.015870639 +0000 UTC m=+920.620178192" watchObservedRunningTime="2026-10-30 22:05:43.016784512 +0000 UTC m=+920.621092066"
Oct 30 22:05:43.787825 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:05:43.787793    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-7bb1c2/simulator-w3-7bb1c2-kserve-675645869c-vcwqv"
Oct 30 22:05:51.788361 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:05:51.788323    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-7bb1c2/simulator-w3-7bb1c2-kserve-675645869c-vcwqv"
Oct 30 22:05:51.800793 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:05:51.800769    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-7bb1c2/simulator-w3-7bb1c2-kserve-675645869c-vcwqv"
Oct 30 22:05:51.955533 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:05:51.955506    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-7bb1c2/simulator-w3-7bb1c2-kserve-675645869c-vcwqv"
Oct 30 22:05:54.249377 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:05:54.249343    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-65e59f/simulator-w4-65e59f-kserve-85698bcd47-przch"]
Oct 30 22:05:54.252515 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:05:54.252492    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-65e59f/simulator-w4-65e59f-kserve-85698bcd47-przch"
Oct 30 22:05:54.314923 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:05:54.314893    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-65e59f\"/\"kube-root-ca.crt\""
Oct 30 22:05:54.348919 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:05:54.348887    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-65e59f\"/\"openshift-service-ca.crt\""
Oct 30 22:05:54.348919 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:05:54.348911    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-65e59f\"/\"simulator-w4-65e59f-kserve-self-signed-certs\""
Oct 30 22:05:54.348919 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:05:54.348908    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-65e59f\"/\"default-dockercfg-zpz5r\""
Oct 30 22:05:54.351482 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:05:54.351459    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-65e59f/simulator-w4-65e59f-kserve-85698bcd47-przch"]
Oct 30 22:05:54.397195 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:05:54.397162    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee5e843b-17be-4623-8c95-dc07c8b7f506-tmp-dir\") pod \"simulator-w4-65e59f-kserve-85698bcd47-przch\" (UID: \"ee5e843b-17be-4623-8c95-dc07c8b7f506\") " pod="e2e-models-e2e-worker-w4-65e59f/simulator-w4-65e59f-kserve-85698bcd47-przch"
Oct 30 22:05:54.397417 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:05:54.397308    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee5e843b-17be-4623-8c95-dc07c8b7f506-tls-certs\") pod \"simulator-w4-65e59f-kserve-85698bcd47-przch\" (UID: \"ee5e843b-17be-4623-8c95-dc07c8b7f506\") " pod="e2e-models-e2e-worker-w4-65e59f/simulator-w4-65e59f-kserve-85698bcd47-przch"
Oct 30 22:05:54.397519 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:05:54.397485    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqq92\" (UniqueName: \"kubernetes.io/projected/ee5e843b-17be-4623-8c95-dc07c8b7f506-kube-api-access-fqq92\") pod \"simulator-w4-65e59f-kserve-85698bcd47-przch\" (UID: \"ee5e843b-17be-4623-8c95-dc07c8b7f506\") " pod="e2e-models-e2e-worker-w4-65e59f/simulator-w4-65e59f-kserve-85698bcd47-przch"
Oct 30 22:05:54.397585 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:05:54.397539    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee5e843b-17be-4623-8c95-dc07c8b7f506-home\") pod \"simulator-w4-65e59f-kserve-85698bcd47-przch\" (UID: \"ee5e843b-17be-4623-8c95-dc07c8b7f506\") " pod="e2e-models-e2e-worker-w4-65e59f/simulator-w4-65e59f-kserve-85698bcd47-przch"
Oct 30 22:05:54.397585 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:05:54.397565    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee5e843b-17be-4623-8c95-dc07c8b7f506-model-cache\") pod \"simulator-w4-65e59f-kserve-85698bcd47-przch\" (UID: \"ee5e843b-17be-4623-8c95-dc07c8b7f506\") " pod="e2e-models-e2e-worker-w4-65e59f/simulator-w4-65e59f-kserve-85698bcd47-przch"
Oct 30 22:05:54.397681 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:05:54.397642    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee5e843b-17be-4623-8c95-dc07c8b7f506-dshm\") pod \"simulator-w4-65e59f-kserve-85698bcd47-przch\" (UID: \"ee5e843b-17be-4623-8c95-dc07c8b7f506\") " pod="e2e-models-e2e-worker-w4-65e59f/simulator-w4-65e59f-kserve-85698bcd47-przch"
Oct 30 22:05:54.498369 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:05:54.498332    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqq92\" (UniqueName: \"kubernetes.io/projected/ee5e843b-17be-4623-8c95-dc07c8b7f506-kube-api-access-fqq92\") pod \"simulator-w4-65e59f-kserve-85698bcd47-przch\" (UID: \"ee5e843b-17be-4623-8c95-dc07c8b7f506\") " pod="e2e-models-e2e-worker-w4-65e59f/simulator-w4-65e59f-kserve-85698bcd47-przch"
Oct 30 22:05:54.498369 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:05:54.498371    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee5e843b-17be-4623-8c95-dc07c8b7f506-home\") pod \"simulator-w4-65e59f-kserve-85698bcd47-przch\" (UID: \"ee5e843b-17be-4623-8c95-dc07c8b7f506\") " pod="e2e-models-e2e-worker-w4-65e59f/simulator-w4-65e59f-kserve-85698bcd47-przch"
Oct 30 22:05:54.498638 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:05:54.498390    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee5e843b-17be-4623-8c95-dc07c8b7f506-model-cache\") pod \"simulator-w4-65e59f-kserve-85698bcd47-przch\" (UID: \"ee5e843b-17be-4623-8c95-dc07c8b7f506\") " pod="e2e-models-e2e-worker-w4-65e59f/simulator-w4-65e59f-kserve-85698bcd47-przch"
Oct 30 22:05:54.498638 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:05:54.498451    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee5e843b-17be-4623-8c95-dc07c8b7f506-dshm\") pod \"simulator-w4-65e59f-kserve-85698bcd47-przch\" (UID: \"ee5e843b-17be-4623-8c95-dc07c8b7f506\") " pod="e2e-models-e2e-worker-w4-65e59f/simulator-w4-65e59f-kserve-85698bcd47-przch"
Oct 30 22:05:54.498638 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:05:54.498472    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee5e843b-17be-4623-8c95-dc07c8b7f506-tmp-dir\") pod \"simulator-w4-65e59f-kserve-85698bcd47-przch\" (UID: \"ee5e843b-17be-4623-8c95-dc07c8b7f506\") " pod="e2e-models-e2e-worker-w4-65e59f/simulator-w4-65e59f-kserve-85698bcd47-przch"
Oct 30 22:05:54.498638 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:05:54.498528    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee5e843b-17be-4623-8c95-dc07c8b7f506-tls-certs\") pod \"simulator-w4-65e59f-kserve-85698bcd47-przch\" (UID: \"ee5e843b-17be-4623-8c95-dc07c8b7f506\") " pod="e2e-models-e2e-worker-w4-65e59f/simulator-w4-65e59f-kserve-85698bcd47-przch"
Oct 30 22:05:54.498879 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:05:54.498813    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee5e843b-17be-4623-8c95-dc07c8b7f506-home\") pod \"simulator-w4-65e59f-kserve-85698bcd47-przch\" (UID: \"ee5e843b-17be-4623-8c95-dc07c8b7f506\") " pod="e2e-models-e2e-worker-w4-65e59f/simulator-w4-65e59f-kserve-85698bcd47-przch"
Oct 30 22:05:54.498879 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:05:54.498850    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee5e843b-17be-4623-8c95-dc07c8b7f506-model-cache\") pod \"simulator-w4-65e59f-kserve-85698bcd47-przch\" (UID: \"ee5e843b-17be-4623-8c95-dc07c8b7f506\") " pod="e2e-models-e2e-worker-w4-65e59f/simulator-w4-65e59f-kserve-85698bcd47-przch"
Oct 30 22:05:54.498979 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:05:54.498948    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee5e843b-17be-4623-8c95-dc07c8b7f506-tmp-dir\") pod \"simulator-w4-65e59f-kserve-85698bcd47-przch\" (UID: \"ee5e843b-17be-4623-8c95-dc07c8b7f506\") " pod="e2e-models-e2e-worker-w4-65e59f/simulator-w4-65e59f-kserve-85698bcd47-przch"
Oct 30 22:05:54.500740 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:05:54.500681    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee5e843b-17be-4623-8c95-dc07c8b7f506-dshm\") pod \"simulator-w4-65e59f-kserve-85698bcd47-przch\" (UID: \"ee5e843b-17be-4623-8c95-dc07c8b7f506\") " pod="e2e-models-e2e-worker-w4-65e59f/simulator-w4-65e59f-kserve-85698bcd47-przch"
Oct 30 22:05:54.501004 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:05:54.500987    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee5e843b-17be-4623-8c95-dc07c8b7f506-tls-certs\") pod \"simulator-w4-65e59f-kserve-85698bcd47-przch\" (UID: \"ee5e843b-17be-4623-8c95-dc07c8b7f506\") " pod="e2e-models-e2e-worker-w4-65e59f/simulator-w4-65e59f-kserve-85698bcd47-przch"
Oct 30 22:05:54.557114 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:05:54.557073    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqq92\" (UniqueName: \"kubernetes.io/projected/ee5e843b-17be-4623-8c95-dc07c8b7f506-kube-api-access-fqq92\") pod \"simulator-w4-65e59f-kserve-85698bcd47-przch\" (UID: \"ee5e843b-17be-4623-8c95-dc07c8b7f506\") " pod="e2e-models-e2e-worker-w4-65e59f/simulator-w4-65e59f-kserve-85698bcd47-przch"
Oct 30 22:05:54.564187 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:05:54.564160    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-65e59f/simulator-w4-65e59f-kserve-85698bcd47-przch"
Oct 30 22:05:54.944341 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:05:54.944314    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-65e59f/simulator-w4-65e59f-kserve-85698bcd47-przch"]
Oct 30 22:05:54.946212 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:05:54.946184    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podee5e843b_17be_4623_8c95_dc07c8b7f506.slice/crio-e8560772cd00e0f348be6afbf5fa84c4dac449d462615c38bb0aa966e61aea65 WatchSource:0}: Error finding container e8560772cd00e0f348be6afbf5fa84c4dac449d462615c38bb0aa966e61aea65: Status 404 returned error can't find the container with id e8560772cd00e0f348be6afbf5fa84c4dac449d462615c38bb0aa966e61aea65
Oct 30 22:05:54.962141 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:05:54.962116    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-65e59f/simulator-w4-65e59f-kserve-85698bcd47-przch" event={"ID":"ee5e843b-17be-4623-8c95-dc07c8b7f506","Type":"ContainerStarted","Data":"e8560772cd00e0f348be6afbf5fa84c4dac449d462615c38bb0aa966e61aea65"}
Oct 30 22:05:55.969532 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:05:55.969495    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-65e59f/simulator-w4-65e59f-kserve-85698bcd47-przch" event={"ID":"ee5e843b-17be-4623-8c95-dc07c8b7f506","Type":"ContainerStarted","Data":"bea563ed6f19fd1ea6443895748dd6fd1c575a1cac8de515cfde7bb974e3c2dd"}
Oct 30 22:05:56.082429 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:05:56.082369    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-65e59f/simulator-w4-65e59f-kserve-85698bcd47-przch" podStartSLOduration=2.082357217 podStartE2EDuration="2.082357217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:05:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:05:56.080453353 +0000 UTC m=+933.684760906" watchObservedRunningTime="2026-10-30 22:05:56.082357217 +0000 UTC m=+933.686664807"
Oct 30 22:05:56.564285 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:05:56.564257    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-65e59f/simulator-w4-65e59f-kserve-85698bcd47-przch"
Oct 30 22:06:04.564350 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:06:04.564318    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-65e59f/simulator-w4-65e59f-kserve-85698bcd47-przch"
Oct 30 22:06:04.576611 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:06:04.576587    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-65e59f/simulator-w4-65e59f-kserve-85698bcd47-przch"
Oct 30 22:06:05.038995 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:06:05.038962    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-65e59f/simulator-w4-65e59f-kserve-85698bcd47-przch"
Oct 30 22:06:10.574656 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:06:10.574610    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 22:06:17.032851 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:06:17.032816    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-779f32/premium-w0-779f32-kserve-67978797c8-jpkvl"]
Oct 30 22:06:17.036260 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:06:17.036237    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-779f32/premium-w0-779f32-kserve-67978797c8-jpkvl"
Oct 30 22:06:17.131629 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:06:17.131592    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhrlr\" (UniqueName: \"kubernetes.io/projected/40138fb9-9a75-4458-a351-11eb61c4a1e3-kube-api-access-mhrlr\") pod \"premium-w0-779f32-kserve-67978797c8-jpkvl\" (UID: \"40138fb9-9a75-4458-a351-11eb61c4a1e3\") " pod="e2e-models-e2e-worker-w0-779f32/premium-w0-779f32-kserve-67978797c8-jpkvl"
Oct 30 22:06:17.131629 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:06:17.131633    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40138fb9-9a75-4458-a351-11eb61c4a1e3-dshm\") pod \"premium-w0-779f32-kserve-67978797c8-jpkvl\" (UID: \"40138fb9-9a75-4458-a351-11eb61c4a1e3\") " pod="e2e-models-e2e-worker-w0-779f32/premium-w0-779f32-kserve-67978797c8-jpkvl"
Oct 30 22:06:17.131819 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:06:17.131656    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40138fb9-9a75-4458-a351-11eb61c4a1e3-model-cache\") pod \"premium-w0-779f32-kserve-67978797c8-jpkvl\" (UID: \"40138fb9-9a75-4458-a351-11eb61c4a1e3\") " pod="e2e-models-e2e-worker-w0-779f32/premium-w0-779f32-kserve-67978797c8-jpkvl"
Oct 30 22:06:17.131819 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:06:17.131775    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40138fb9-9a75-4458-a351-11eb61c4a1e3-home\") pod \"premium-w0-779f32-kserve-67978797c8-jpkvl\" (UID: \"40138fb9-9a75-4458-a351-11eb61c4a1e3\") " pod="e2e-models-e2e-worker-w0-779f32/premium-w0-779f32-kserve-67978797c8-jpkvl"
Oct 30 22:06:17.131883 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:06:17.131823    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40138fb9-9a75-4458-a351-11eb61c4a1e3-tmp-dir\") pod \"premium-w0-779f32-kserve-67978797c8-jpkvl\" (UID: \"40138fb9-9a75-4458-a351-11eb61c4a1e3\") " pod="e2e-models-e2e-worker-w0-779f32/premium-w0-779f32-kserve-67978797c8-jpkvl"
Oct 30 22:06:17.131883 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:06:17.131861    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40138fb9-9a75-4458-a351-11eb61c4a1e3-tls-certs\") pod \"premium-w0-779f32-kserve-67978797c8-jpkvl\" (UID: \"40138fb9-9a75-4458-a351-11eb61c4a1e3\") " pod="e2e-models-e2e-worker-w0-779f32/premium-w0-779f32-kserve-67978797c8-jpkvl"
Oct 30 22:06:17.135779 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:06:17.135756    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-779f32\"/\"default-dockercfg-lcntz\""
Oct 30 22:06:17.135779 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:06:17.135782    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-779f32\"/\"kube-root-ca.crt\""
Oct 30 22:06:17.135928 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:06:17.135824    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-779f32\"/\"premium-w0-779f32-kserve-self-signed-certs\""
Oct 30 22:06:17.135928 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:06:17.135842    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-779f32\"/\"openshift-service-ca.crt\""
Oct 30 22:06:17.138535 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:06:17.138516    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-779f32/premium-w0-779f32-kserve-67978797c8-jpkvl"]
Oct 30 22:06:17.233479 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:06:17.233444    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40138fb9-9a75-4458-a351-11eb61c4a1e3-tls-certs\") pod \"premium-w0-779f32-kserve-67978797c8-jpkvl\" (UID: \"40138fb9-9a75-4458-a351-11eb61c4a1e3\") " pod="e2e-models-e2e-worker-w0-779f32/premium-w0-779f32-kserve-67978797c8-jpkvl"
Oct 30 22:06:17.233684 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:06:17.233531    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhrlr\" (UniqueName: \"kubernetes.io/projected/40138fb9-9a75-4458-a351-11eb61c4a1e3-kube-api-access-mhrlr\") pod \"premium-w0-779f32-kserve-67978797c8-jpkvl\" (UID: \"40138fb9-9a75-4458-a351-11eb61c4a1e3\") " pod="e2e-models-e2e-worker-w0-779f32/premium-w0-779f32-kserve-67978797c8-jpkvl"
Oct 30 22:06:17.233684 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:06:17.233559    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40138fb9-9a75-4458-a351-11eb61c4a1e3-dshm\") pod \"premium-w0-779f32-kserve-67978797c8-jpkvl\" (UID: \"40138fb9-9a75-4458-a351-11eb61c4a1e3\") " pod="e2e-models-e2e-worker-w0-779f32/premium-w0-779f32-kserve-67978797c8-jpkvl"
Oct 30 22:06:17.233684 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:06:17.233583    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40138fb9-9a75-4458-a351-11eb61c4a1e3-model-cache\") pod \"premium-w0-779f32-kserve-67978797c8-jpkvl\" (UID: \"40138fb9-9a75-4458-a351-11eb61c4a1e3\") " pod="e2e-models-e2e-worker-w0-779f32/premium-w0-779f32-kserve-67978797c8-jpkvl"
Oct 30 22:06:17.233684 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:06:17.233617    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40138fb9-9a75-4458-a351-11eb61c4a1e3-home\") pod \"premium-w0-779f32-kserve-67978797c8-jpkvl\" (UID: \"40138fb9-9a75-4458-a351-11eb61c4a1e3\") " pod="e2e-models-e2e-worker-w0-779f32/premium-w0-779f32-kserve-67978797c8-jpkvl"
Oct 30 22:06:17.233684 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:06:17.233679    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40138fb9-9a75-4458-a351-11eb61c4a1e3-tmp-dir\") pod \"premium-w0-779f32-kserve-67978797c8-jpkvl\" (UID: \"40138fb9-9a75-4458-a351-11eb61c4a1e3\") " pod="e2e-models-e2e-worker-w0-779f32/premium-w0-779f32-kserve-67978797c8-jpkvl"
Oct 30 22:06:17.234045 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:06:17.234015    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40138fb9-9a75-4458-a351-11eb61c4a1e3-model-cache\") pod \"premium-w0-779f32-kserve-67978797c8-jpkvl\" (UID: \"40138fb9-9a75-4458-a351-11eb61c4a1e3\") " pod="e2e-models-e2e-worker-w0-779f32/premium-w0-779f32-kserve-67978797c8-jpkvl"
Oct 30 22:06:17.234179 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:06:17.234139    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40138fb9-9a75-4458-a351-11eb61c4a1e3-home\") pod \"premium-w0-779f32-kserve-67978797c8-jpkvl\" (UID: \"40138fb9-9a75-4458-a351-11eb61c4a1e3\") " pod="e2e-models-e2e-worker-w0-779f32/premium-w0-779f32-kserve-67978797c8-jpkvl"
Oct 30 22:06:17.234179 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:06:17.234152    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40138fb9-9a75-4458-a351-11eb61c4a1e3-tmp-dir\") pod \"premium-w0-779f32-kserve-67978797c8-jpkvl\" (UID: \"40138fb9-9a75-4458-a351-11eb61c4a1e3\") " pod="e2e-models-e2e-worker-w0-779f32/premium-w0-779f32-kserve-67978797c8-jpkvl"
Oct 30 22:06:17.235836 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:06:17.235809    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40138fb9-9a75-4458-a351-11eb61c4a1e3-dshm\") pod \"premium-w0-779f32-kserve-67978797c8-jpkvl\" (UID: \"40138fb9-9a75-4458-a351-11eb61c4a1e3\") " pod="e2e-models-e2e-worker-w0-779f32/premium-w0-779f32-kserve-67978797c8-jpkvl"
Oct 30 22:06:17.236042 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:06:17.236025    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40138fb9-9a75-4458-a351-11eb61c4a1e3-tls-certs\") pod \"premium-w0-779f32-kserve-67978797c8-jpkvl\" (UID: \"40138fb9-9a75-4458-a351-11eb61c4a1e3\") " pod="e2e-models-e2e-worker-w0-779f32/premium-w0-779f32-kserve-67978797c8-jpkvl"
Oct 30 22:06:17.291666 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:06:17.291599    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhrlr\" (UniqueName: \"kubernetes.io/projected/40138fb9-9a75-4458-a351-11eb61c4a1e3-kube-api-access-mhrlr\") pod \"premium-w0-779f32-kserve-67978797c8-jpkvl\" (UID: \"40138fb9-9a75-4458-a351-11eb61c4a1e3\") " pod="e2e-models-e2e-worker-w0-779f32/premium-w0-779f32-kserve-67978797c8-jpkvl"
Oct 30 22:06:17.349268 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:06:17.349240    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-779f32/premium-w0-779f32-kserve-67978797c8-jpkvl"
Oct 30 22:06:17.750939 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:06:17.750905    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-779f32/premium-w0-779f32-kserve-67978797c8-jpkvl"]
Oct 30 22:06:17.752496 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:06:17.752466    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod40138fb9_9a75_4458_a351_11eb61c4a1e3.slice/crio-555c56d6f5ba3296e5487c55eb769cadb6a650dda1607fe4fb6de0bcb45190e7 WatchSource:0}: Error finding container 555c56d6f5ba3296e5487c55eb769cadb6a650dda1607fe4fb6de0bcb45190e7: Status 404 returned error can't find the container with id 555c56d6f5ba3296e5487c55eb769cadb6a650dda1607fe4fb6de0bcb45190e7
Oct 30 22:06:18.103995 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:06:18.103906    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-779f32/premium-w0-779f32-kserve-67978797c8-jpkvl" event={"ID":"40138fb9-9a75-4458-a351-11eb61c4a1e3","Type":"ContainerStarted","Data":"f54bcac0d9e7ceb682de4e2c27e8d08fe11ac442a95ca35ad437b2acc431cac0"}
Oct 30 22:06:18.103995 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:06:18.103953    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-779f32/premium-w0-779f32-kserve-67978797c8-jpkvl" event={"ID":"40138fb9-9a75-4458-a351-11eb61c4a1e3","Type":"ContainerStarted","Data":"555c56d6f5ba3296e5487c55eb769cadb6a650dda1607fe4fb6de0bcb45190e7"}
Oct 30 22:06:18.216932 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:06:18.216873    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-779f32/premium-w0-779f32-kserve-67978797c8-jpkvl" podStartSLOduration=2.216860037 podStartE2EDuration="2.216860037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:06:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:18.215740984 +0000 UTC m=+955.820048535" watchObservedRunningTime="2026-10-30 22:06:18.216860037 +0000 UTC m=+955.821167648"
Oct 30 22:06:18.349848 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:06:18.349812    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-779f32/premium-w0-779f32-kserve-67978797c8-jpkvl"
Oct 30 22:06:27.349769 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:06:27.349729    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-779f32/premium-w0-779f32-kserve-67978797c8-jpkvl"
Oct 30 22:06:27.361957 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:06:27.361935    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-779f32/premium-w0-779f32-kserve-67978797c8-jpkvl"
Oct 30 22:06:28.169839 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:06:28.169810    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-779f32/premium-w0-779f32-kserve-67978797c8-jpkvl"
Oct 30 22:06:40.571104 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:06:40.571068    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 22:07:08.742299 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:07:08.742216    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-7bb1c2/premium-w3-7bb1c2-kserve-7dbf875d88-dqxn5"]
Oct 30 22:07:08.745540 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:07:08.745517    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-7bb1c2/premium-w3-7bb1c2-kserve-7dbf875d88-dqxn5"
Oct 30 22:07:08.830717 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:07:08.830679    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a58c911f-e2eb-4820-a6b6-24afe78b5fd4-model-cache\") pod \"premium-w3-7bb1c2-kserve-7dbf875d88-dqxn5\" (UID: \"a58c911f-e2eb-4820-a6b6-24afe78b5fd4\") " pod="e2e-models-e2e-worker-w3-7bb1c2/premium-w3-7bb1c2-kserve-7dbf875d88-dqxn5"
Oct 30 22:07:08.830877 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:07:08.830799    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a58c911f-e2eb-4820-a6b6-24afe78b5fd4-tls-certs\") pod \"premium-w3-7bb1c2-kserve-7dbf875d88-dqxn5\" (UID: \"a58c911f-e2eb-4820-a6b6-24afe78b5fd4\") " pod="e2e-models-e2e-worker-w3-7bb1c2/premium-w3-7bb1c2-kserve-7dbf875d88-dqxn5"
Oct 30 22:07:08.830877 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:07:08.830829    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a58c911f-e2eb-4820-a6b6-24afe78b5fd4-tmp-dir\") pod \"premium-w3-7bb1c2-kserve-7dbf875d88-dqxn5\" (UID: \"a58c911f-e2eb-4820-a6b6-24afe78b5fd4\") " pod="e2e-models-e2e-worker-w3-7bb1c2/premium-w3-7bb1c2-kserve-7dbf875d88-dqxn5"
Oct 30 22:07:08.830877 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:07:08.830849    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a58c911f-e2eb-4820-a6b6-24afe78b5fd4-dshm\") pod \"premium-w3-7bb1c2-kserve-7dbf875d88-dqxn5\" (UID: \"a58c911f-e2eb-4820-a6b6-24afe78b5fd4\") " pod="e2e-models-e2e-worker-w3-7bb1c2/premium-w3-7bb1c2-kserve-7dbf875d88-dqxn5"
Oct 30 22:07:08.831079 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:07:08.830919    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvft7\" (UniqueName: \"kubernetes.io/projected/a58c911f-e2eb-4820-a6b6-24afe78b5fd4-kube-api-access-xvft7\") pod \"premium-w3-7bb1c2-kserve-7dbf875d88-dqxn5\" (UID: \"a58c911f-e2eb-4820-a6b6-24afe78b5fd4\") " pod="e2e-models-e2e-worker-w3-7bb1c2/premium-w3-7bb1c2-kserve-7dbf875d88-dqxn5"
Oct 30 22:07:08.831136 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:07:08.831090    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a58c911f-e2eb-4820-a6b6-24afe78b5fd4-home\") pod \"premium-w3-7bb1c2-kserve-7dbf875d88-dqxn5\" (UID: \"a58c911f-e2eb-4820-a6b6-24afe78b5fd4\") " pod="e2e-models-e2e-worker-w3-7bb1c2/premium-w3-7bb1c2-kserve-7dbf875d88-dqxn5"
Oct 30 22:07:08.842416 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:07:08.842376    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-7bb1c2\"/\"premium-w3-7bb1c2-kserve-self-signed-certs\""
Oct 30 22:07:08.844876 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:07:08.844858    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-7bb1c2/premium-w3-7bb1c2-kserve-7dbf875d88-dqxn5"]
Oct 30 22:07:08.932302 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:07:08.932266    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a58c911f-e2eb-4820-a6b6-24afe78b5fd4-model-cache\") pod \"premium-w3-7bb1c2-kserve-7dbf875d88-dqxn5\" (UID: \"a58c911f-e2eb-4820-a6b6-24afe78b5fd4\") " pod="e2e-models-e2e-worker-w3-7bb1c2/premium-w3-7bb1c2-kserve-7dbf875d88-dqxn5"
Oct 30 22:07:08.932529 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:07:08.932368    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a58c911f-e2eb-4820-a6b6-24afe78b5fd4-tls-certs\") pod \"premium-w3-7bb1c2-kserve-7dbf875d88-dqxn5\" (UID: \"a58c911f-e2eb-4820-a6b6-24afe78b5fd4\") " pod="e2e-models-e2e-worker-w3-7bb1c2/premium-w3-7bb1c2-kserve-7dbf875d88-dqxn5"
Oct 30 22:07:08.932529 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:07:08.932424    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a58c911f-e2eb-4820-a6b6-24afe78b5fd4-tmp-dir\") pod \"premium-w3-7bb1c2-kserve-7dbf875d88-dqxn5\" (UID: \"a58c911f-e2eb-4820-a6b6-24afe78b5fd4\") " pod="e2e-models-e2e-worker-w3-7bb1c2/premium-w3-7bb1c2-kserve-7dbf875d88-dqxn5"
Oct 30 22:07:08.932529 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:07:08.932457    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a58c911f-e2eb-4820-a6b6-24afe78b5fd4-dshm\") pod \"premium-w3-7bb1c2-kserve-7dbf875d88-dqxn5\" (UID: \"a58c911f-e2eb-4820-a6b6-24afe78b5fd4\") " pod="e2e-models-e2e-worker-w3-7bb1c2/premium-w3-7bb1c2-kserve-7dbf875d88-dqxn5"
Oct 30 22:07:08.932529 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:07:08.932494    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvft7\" (UniqueName: \"kubernetes.io/projected/a58c911f-e2eb-4820-a6b6-24afe78b5fd4-kube-api-access-xvft7\") pod \"premium-w3-7bb1c2-kserve-7dbf875d88-dqxn5\" (UID: \"a58c911f-e2eb-4820-a6b6-24afe78b5fd4\") " pod="e2e-models-e2e-worker-w3-7bb1c2/premium-w3-7bb1c2-kserve-7dbf875d88-dqxn5"
Oct 30 22:07:08.932751 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:07:08.932547    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a58c911f-e2eb-4820-a6b6-24afe78b5fd4-home\") pod \"premium-w3-7bb1c2-kserve-7dbf875d88-dqxn5\" (UID: \"a58c911f-e2eb-4820-a6b6-24afe78b5fd4\") " pod="e2e-models-e2e-worker-w3-7bb1c2/premium-w3-7bb1c2-kserve-7dbf875d88-dqxn5"
Oct 30 22:07:08.932751 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:07:08.932734    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a58c911f-e2eb-4820-a6b6-24afe78b5fd4-model-cache\") pod \"premium-w3-7bb1c2-kserve-7dbf875d88-dqxn5\" (UID: \"a58c911f-e2eb-4820-a6b6-24afe78b5fd4\") " pod="e2e-models-e2e-worker-w3-7bb1c2/premium-w3-7bb1c2-kserve-7dbf875d88-dqxn5"
Oct 30 22:07:08.932873 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:07:08.932821    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a58c911f-e2eb-4820-a6b6-24afe78b5fd4-tmp-dir\") pod \"premium-w3-7bb1c2-kserve-7dbf875d88-dqxn5\" (UID: \"a58c911f-e2eb-4820-a6b6-24afe78b5fd4\") " pod="e2e-models-e2e-worker-w3-7bb1c2/premium-w3-7bb1c2-kserve-7dbf875d88-dqxn5"
Oct 30 22:07:08.932959 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:07:08.932940    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a58c911f-e2eb-4820-a6b6-24afe78b5fd4-home\") pod \"premium-w3-7bb1c2-kserve-7dbf875d88-dqxn5\" (UID: \"a58c911f-e2eb-4820-a6b6-24afe78b5fd4\") " pod="e2e-models-e2e-worker-w3-7bb1c2/premium-w3-7bb1c2-kserve-7dbf875d88-dqxn5"
Oct 30 22:07:08.934718 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:07:08.934697    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a58c911f-e2eb-4820-a6b6-24afe78b5fd4-dshm\") pod \"premium-w3-7bb1c2-kserve-7dbf875d88-dqxn5\" (UID: \"a58c911f-e2eb-4820-a6b6-24afe78b5fd4\") " pod="e2e-models-e2e-worker-w3-7bb1c2/premium-w3-7bb1c2-kserve-7dbf875d88-dqxn5"
Oct 30 22:07:08.934986 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:07:08.934969    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a58c911f-e2eb-4820-a6b6-24afe78b5fd4-tls-certs\") pod \"premium-w3-7bb1c2-kserve-7dbf875d88-dqxn5\" (UID: \"a58c911f-e2eb-4820-a6b6-24afe78b5fd4\") " pod="e2e-models-e2e-worker-w3-7bb1c2/premium-w3-7bb1c2-kserve-7dbf875d88-dqxn5"
Oct 30 22:07:08.994191 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:07:08.994113    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvft7\" (UniqueName: \"kubernetes.io/projected/a58c911f-e2eb-4820-a6b6-24afe78b5fd4-kube-api-access-xvft7\") pod \"premium-w3-7bb1c2-kserve-7dbf875d88-dqxn5\" (UID: \"a58c911f-e2eb-4820-a6b6-24afe78b5fd4\") " pod="e2e-models-e2e-worker-w3-7bb1c2/premium-w3-7bb1c2-kserve-7dbf875d88-dqxn5"
Oct 30 22:07:09.057076 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:07:09.057040    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-7bb1c2/premium-w3-7bb1c2-kserve-7dbf875d88-dqxn5"
Oct 30 22:07:09.434295 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:07:09.434266    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-7bb1c2/premium-w3-7bb1c2-kserve-7dbf875d88-dqxn5"]
Oct 30 22:07:09.435641 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:07:09.435615    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda58c911f_e2eb_4820_a6b6_24afe78b5fd4.slice/crio-7db2ee57a6bc5b2ab0c3b1f82c03272077341767e8dd93b3a1eb8fdac07a848f WatchSource:0}: Error finding container 7db2ee57a6bc5b2ab0c3b1f82c03272077341767e8dd93b3a1eb8fdac07a848f: Status 404 returned error can't find the container with id 7db2ee57a6bc5b2ab0c3b1f82c03272077341767e8dd93b3a1eb8fdac07a848f
Oct 30 22:07:10.417122 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:07:10.417091    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-7bb1c2/premium-w3-7bb1c2-kserve-7dbf875d88-dqxn5" event={"ID":"a58c911f-e2eb-4820-a6b6-24afe78b5fd4","Type":"ContainerStarted","Data":"847e616c8cb13638a02400802170fbea1247c1ac01ba3d3d06107f923c6060fd"}
Oct 30 22:07:10.417527 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:07:10.417131    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-7bb1c2/premium-w3-7bb1c2-kserve-7dbf875d88-dqxn5" event={"ID":"a58c911f-e2eb-4820-a6b6-24afe78b5fd4","Type":"ContainerStarted","Data":"7db2ee57a6bc5b2ab0c3b1f82c03272077341767e8dd93b3a1eb8fdac07a848f"}
Oct 30 22:07:10.531634 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:07:10.531580    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-7bb1c2/premium-w3-7bb1c2-kserve-7dbf875d88-dqxn5" podStartSLOduration=2.531566815 podStartE2EDuration="2.531566815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:07:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:10.529775879 +0000 UTC m=+1008.134083437" watchObservedRunningTime="2026-10-30 22:07:10.531566815 +0000 UTC m=+1008.135874398"
Oct 30 22:07:10.571225 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:07:10.571195    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 22:07:11.057264 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:07:11.057231    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-7bb1c2/premium-w3-7bb1c2-kserve-7dbf875d88-dqxn5"
Oct 30 22:07:19.057842 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:07:19.057803    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-7bb1c2/premium-w3-7bb1c2-kserve-7dbf875d88-dqxn5"
Oct 30 22:07:19.070434 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:07:19.070383    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-7bb1c2/premium-w3-7bb1c2-kserve-7dbf875d88-dqxn5"
Oct 30 22:07:19.488392 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:07:19.488356    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-7bb1c2/premium-w3-7bb1c2-kserve-7dbf875d88-dqxn5"
Oct 30 22:07:19.897007 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:07:19.896982    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-65e59f/premium-w4-65e59f-kserve-6dc49468f8-rlzhx"]
Oct 30 22:07:19.900194 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:07:19.900173    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-65e59f/premium-w4-65e59f-kserve-6dc49468f8-rlzhx"
Oct 30 22:07:19.949945 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:07:19.949908    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-65e59f\"/\"premium-w4-65e59f-kserve-self-signed-certs\""
Oct 30 22:07:19.965112 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:07:19.965080    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8590a783-eb24-4759-ad74-1369f36aa9d2-model-cache\") pod \"premium-w4-65e59f-kserve-6dc49468f8-rlzhx\" (UID: \"8590a783-eb24-4759-ad74-1369f36aa9d2\") " pod="e2e-models-e2e-worker-w4-65e59f/premium-w4-65e59f-kserve-6dc49468f8-rlzhx"
Oct 30 22:07:19.965255 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:07:19.965131    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8590a783-eb24-4759-ad74-1369f36aa9d2-tmp-dir\") pod \"premium-w4-65e59f-kserve-6dc49468f8-rlzhx\" (UID: \"8590a783-eb24-4759-ad74-1369f36aa9d2\") " pod="e2e-models-e2e-worker-w4-65e59f/premium-w4-65e59f-kserve-6dc49468f8-rlzhx"
Oct 30 22:07:19.965255 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:07:19.965210    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8590a783-eb24-4759-ad74-1369f36aa9d2-home\") pod \"premium-w4-65e59f-kserve-6dc49468f8-rlzhx\" (UID: \"8590a783-eb24-4759-ad74-1369f36aa9d2\") " pod="e2e-models-e2e-worker-w4-65e59f/premium-w4-65e59f-kserve-6dc49468f8-rlzhx"
Oct 30 22:07:19.965351 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:07:19.965334    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xfxz\" (UniqueName: \"kubernetes.io/projected/8590a783-eb24-4759-ad74-1369f36aa9d2-kube-api-access-6xfxz\") pod \"premium-w4-65e59f-kserve-6dc49468f8-rlzhx\" (UID: \"8590a783-eb24-4759-ad74-1369f36aa9d2\") " pod="e2e-models-e2e-worker-w4-65e59f/premium-w4-65e59f-kserve-6dc49468f8-rlzhx"
Oct 30 22:07:19.965392 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:07:19.965375    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8590a783-eb24-4759-ad74-1369f36aa9d2-dshm\") pod \"premium-w4-65e59f-kserve-6dc49468f8-rlzhx\" (UID: \"8590a783-eb24-4759-ad74-1369f36aa9d2\") " pod="e2e-models-e2e-worker-w4-65e59f/premium-w4-65e59f-kserve-6dc49468f8-rlzhx"
Oct 30 22:07:19.965531 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:07:19.965505    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8590a783-eb24-4759-ad74-1369f36aa9d2-tls-certs\") pod \"premium-w4-65e59f-kserve-6dc49468f8-rlzhx\" (UID: \"8590a783-eb24-4759-ad74-1369f36aa9d2\") " pod="e2e-models-e2e-worker-w4-65e59f/premium-w4-65e59f-kserve-6dc49468f8-rlzhx"
Oct 30 22:07:19.999151 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:07:19.999115    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-65e59f/premium-w4-65e59f-kserve-6dc49468f8-rlzhx"]
Oct 30 22:07:20.066235 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:07:20.066197    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8590a783-eb24-4759-ad74-1369f36aa9d2-model-cache\") pod \"premium-w4-65e59f-kserve-6dc49468f8-rlzhx\" (UID: \"8590a783-eb24-4759-ad74-1369f36aa9d2\") " pod="e2e-models-e2e-worker-w4-65e59f/premium-w4-65e59f-kserve-6dc49468f8-rlzhx"
Oct 30 22:07:20.066769 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:07:20.066264    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8590a783-eb24-4759-ad74-1369f36aa9d2-tmp-dir\") pod \"premium-w4-65e59f-kserve-6dc49468f8-rlzhx\" (UID: \"8590a783-eb24-4759-ad74-1369f36aa9d2\") " pod="e2e-models-e2e-worker-w4-65e59f/premium-w4-65e59f-kserve-6dc49468f8-rlzhx"
Oct 30 22:07:20.066769 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:07:20.066300    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8590a783-eb24-4759-ad74-1369f36aa9d2-home\") pod \"premium-w4-65e59f-kserve-6dc49468f8-rlzhx\" (UID: \"8590a783-eb24-4759-ad74-1369f36aa9d2\") " pod="e2e-models-e2e-worker-w4-65e59f/premium-w4-65e59f-kserve-6dc49468f8-rlzhx"
Oct 30 22:07:20.066769 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:07:20.066358    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xfxz\" (UniqueName: \"kubernetes.io/projected/8590a783-eb24-4759-ad74-1369f36aa9d2-kube-api-access-6xfxz\") pod \"premium-w4-65e59f-kserve-6dc49468f8-rlzhx\" (UID: \"8590a783-eb24-4759-ad74-1369f36aa9d2\") " pod="e2e-models-e2e-worker-w4-65e59f/premium-w4-65e59f-kserve-6dc49468f8-rlzhx"
Oct 30 22:07:20.066769 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:07:20.066394    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8590a783-eb24-4759-ad74-1369f36aa9d2-dshm\") pod \"premium-w4-65e59f-kserve-6dc49468f8-rlzhx\" (UID: \"8590a783-eb24-4759-ad74-1369f36aa9d2\") " pod="e2e-models-e2e-worker-w4-65e59f/premium-w4-65e59f-kserve-6dc49468f8-rlzhx"
Oct 30 22:07:20.066769 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:07:20.066448    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8590a783-eb24-4759-ad74-1369f36aa9d2-tls-certs\") pod \"premium-w4-65e59f-kserve-6dc49468f8-rlzhx\" (UID: \"8590a783-eb24-4759-ad74-1369f36aa9d2\") " pod="e2e-models-e2e-worker-w4-65e59f/premium-w4-65e59f-kserve-6dc49468f8-rlzhx"
Oct 30 22:07:20.066769 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:07:20.066613    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8590a783-eb24-4759-ad74-1369f36aa9d2-model-cache\") pod \"premium-w4-65e59f-kserve-6dc49468f8-rlzhx\" (UID: \"8590a783-eb24-4759-ad74-1369f36aa9d2\") " pod="e2e-models-e2e-worker-w4-65e59f/premium-w4-65e59f-kserve-6dc49468f8-rlzhx"
Oct 30 22:07:20.066769 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:07:20.066667    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8590a783-eb24-4759-ad74-1369f36aa9d2-tmp-dir\") pod \"premium-w4-65e59f-kserve-6dc49468f8-rlzhx\" (UID: \"8590a783-eb24-4759-ad74-1369f36aa9d2\") " pod="e2e-models-e2e-worker-w4-65e59f/premium-w4-65e59f-kserve-6dc49468f8-rlzhx"
Oct 30 22:07:20.066769 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:07:20.066737    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8590a783-eb24-4759-ad74-1369f36aa9d2-home\") pod \"premium-w4-65e59f-kserve-6dc49468f8-rlzhx\" (UID: \"8590a783-eb24-4759-ad74-1369f36aa9d2\") " pod="e2e-models-e2e-worker-w4-65e59f/premium-w4-65e59f-kserve-6dc49468f8-rlzhx"
Oct 30 22:07:20.068732 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:07:20.068711    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8590a783-eb24-4759-ad74-1369f36aa9d2-dshm\") pod \"premium-w4-65e59f-kserve-6dc49468f8-rlzhx\" (UID: \"8590a783-eb24-4759-ad74-1369f36aa9d2\") " pod="e2e-models-e2e-worker-w4-65e59f/premium-w4-65e59f-kserve-6dc49468f8-rlzhx"
Oct 30 22:07:20.069034 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:07:20.069017    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8590a783-eb24-4759-ad74-1369f36aa9d2-tls-certs\") pod \"premium-w4-65e59f-kserve-6dc49468f8-rlzhx\" (UID: \"8590a783-eb24-4759-ad74-1369f36aa9d2\") " pod="e2e-models-e2e-worker-w4-65e59f/premium-w4-65e59f-kserve-6dc49468f8-rlzhx"
Oct 30 22:07:20.139873 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:07:20.139842    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xfxz\" (UniqueName: \"kubernetes.io/projected/8590a783-eb24-4759-ad74-1369f36aa9d2-kube-api-access-6xfxz\") pod \"premium-w4-65e59f-kserve-6dc49468f8-rlzhx\" (UID: \"8590a783-eb24-4759-ad74-1369f36aa9d2\") " pod="e2e-models-e2e-worker-w4-65e59f/premium-w4-65e59f-kserve-6dc49468f8-rlzhx"
Oct 30 22:07:20.211793 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:07:20.211719    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-65e59f/premium-w4-65e59f-kserve-6dc49468f8-rlzhx"
Oct 30 22:07:20.597249 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:07:20.597220    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-65e59f/premium-w4-65e59f-kserve-6dc49468f8-rlzhx"]
Oct 30 22:07:20.599025 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:07:20.598995    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8590a783_eb24_4759_ad74_1369f36aa9d2.slice/crio-26a0b42d67819a9271e7adaf0b58b3150bcedb414e430aebfcd9c59c44fc9b1d WatchSource:0}: Error finding container 26a0b42d67819a9271e7adaf0b58b3150bcedb414e430aebfcd9c59c44fc9b1d: Status 404 returned error can't find the container with id 26a0b42d67819a9271e7adaf0b58b3150bcedb414e430aebfcd9c59c44fc9b1d
Oct 30 22:07:21.483327 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:07:21.483282    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-65e59f/premium-w4-65e59f-kserve-6dc49468f8-rlzhx" event={"ID":"8590a783-eb24-4759-ad74-1369f36aa9d2","Type":"ContainerStarted","Data":"0e4df1de02bb062c21465c899d86aabb41fe51a6cef551344991550623680d7f"}
Oct 30 22:07:21.483327 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:07:21.483336    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-65e59f/premium-w4-65e59f-kserve-6dc49468f8-rlzhx" event={"ID":"8590a783-eb24-4759-ad74-1369f36aa9d2","Type":"ContainerStarted","Data":"26a0b42d67819a9271e7adaf0b58b3150bcedb414e430aebfcd9c59c44fc9b1d"}
Oct 30 22:07:21.619516 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:07:21.619469    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-65e59f/premium-w4-65e59f-kserve-6dc49468f8-rlzhx" podStartSLOduration=2.619455897 podStartE2EDuration="2.619455897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:07:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:21.617187781 +0000 UTC m=+1019.221495334" watchObservedRunningTime="2026-10-30 22:07:21.619455897 +0000 UTC m=+1019.223763466"
Oct 30 22:07:22.211992 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:07:22.211958    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-65e59f/premium-w4-65e59f-kserve-6dc49468f8-rlzhx"
Oct 30 22:07:30.206250 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:07:30.206215    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ldgt7"]
Oct 30 22:07:30.209603 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:07:30.209586    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ldgt7"
Oct 30 22:07:30.212390 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:07:30.212364    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-65e59f/premium-w4-65e59f-kserve-6dc49468f8-rlzhx"
Oct 30 22:07:30.225392 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:07:30.225371    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-65e59f/premium-w4-65e59f-kserve-6dc49468f8-rlzhx"
Oct 30 22:07:30.259308 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:07:30.259288    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-7bb1c2-openshift-default-dockercfg-9dlcb\""
Oct 30 22:07:30.272030 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:07:30.271997    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e17f2f4-e217-43cf-81ba-2bfb2095dbd6-istiod-ca-cert\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ldgt7\" (UID: \"0e17f2f4-e217-43cf-81ba-2bfb2095dbd6\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ldgt7"
Oct 30 22:07:30.272137 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:07:30.272036    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e17f2f4-e217-43cf-81ba-2bfb2095dbd6-credential-socket\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ldgt7\" (UID: \"0e17f2f4-e217-43cf-81ba-2bfb2095dbd6\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ldgt7"
Oct 30 22:07:30.272137 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:07:30.272061    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e17f2f4-e217-43cf-81ba-2bfb2095dbd6-istio-envoy\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ldgt7\" (UID: \"0e17f2f4-e217-43cf-81ba-2bfb2095dbd6\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ldgt7"
Oct 30 22:07:30.272243 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:07:30.272216    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e17f2f4-e217-43cf-81ba-2bfb2095dbd6-istio-token\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ldgt7\" (UID: \"0e17f2f4-e217-43cf-81ba-2bfb2095dbd6\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ldgt7"
Oct 30 22:07:30.272291 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:07:30.272275    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e17f2f4-e217-43cf-81ba-2bfb2095dbd6-workload-certs\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ldgt7\" (UID: \"0e17f2f4-e217-43cf-81ba-2bfb2095dbd6\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ldgt7"
Oct 30 22:07:30.272475 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:07:30.272453    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e17f2f4-e217-43cf-81ba-2bfb2095dbd6-istio-data\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ldgt7\" (UID: \"0e17f2f4-e217-43cf-81ba-2bfb2095dbd6\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ldgt7"
Oct 30 22:07:30.272566 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:07:30.272490    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e17f2f4-e217-43cf-81ba-2bfb2095dbd6-workload-socket\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ldgt7\" (UID: \"0e17f2f4-e217-43cf-81ba-2bfb2095dbd6\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ldgt7"
Oct 30 22:07:30.272616 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:07:30.272589    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e17f2f4-e217-43cf-81ba-2bfb2095dbd6-istio-podinfo\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ldgt7\" (UID: \"0e17f2f4-e217-43cf-81ba-2bfb2095dbd6\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ldgt7"
Oct 30 22:07:30.272661 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:07:30.272648    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dr7d\" (UniqueName: \"kubernetes.io/projected/0e17f2f4-e217-43cf-81ba-2bfb2095dbd6-kube-api-access-7dr7d\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ldgt7\" (UID: \"0e17f2f4-e217-43cf-81ba-2bfb2095dbd6\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ldgt7"
Oct 30 22:07:30.309311 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:07:30.309283    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ldgt7"]
Oct 30 22:07:30.373567 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:07:30.373537    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e17f2f4-e217-43cf-81ba-2bfb2095dbd6-istiod-ca-cert\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ldgt7\" (UID: \"0e17f2f4-e217-43cf-81ba-2bfb2095dbd6\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ldgt7"
Oct 30 22:07:30.373751 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:07:30.373572    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e17f2f4-e217-43cf-81ba-2bfb2095dbd6-credential-socket\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ldgt7\" (UID: \"0e17f2f4-e217-43cf-81ba-2bfb2095dbd6\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ldgt7"
Oct 30 22:07:30.373751 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:07:30.373599    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e17f2f4-e217-43cf-81ba-2bfb2095dbd6-istio-envoy\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ldgt7\" (UID: \"0e17f2f4-e217-43cf-81ba-2bfb2095dbd6\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ldgt7"
Oct 30 22:07:30.373751 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:07:30.373631    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e17f2f4-e217-43cf-81ba-2bfb2095dbd6-istio-token\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ldgt7\" (UID: \"0e17f2f4-e217-43cf-81ba-2bfb2095dbd6\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ldgt7"
Oct 30 22:07:30.373751 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:07:30.373655    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e17f2f4-e217-43cf-81ba-2bfb2095dbd6-workload-certs\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ldgt7\" (UID: \"0e17f2f4-e217-43cf-81ba-2bfb2095dbd6\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ldgt7"
Oct 30 22:07:30.373751 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:07:30.373680    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e17f2f4-e217-43cf-81ba-2bfb2095dbd6-istio-data\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ldgt7\" (UID: \"0e17f2f4-e217-43cf-81ba-2bfb2095dbd6\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ldgt7"
Oct 30 22:07:30.373751 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:07:30.373704    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e17f2f4-e217-43cf-81ba-2bfb2095dbd6-workload-socket\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ldgt7\" (UID: \"0e17f2f4-e217-43cf-81ba-2bfb2095dbd6\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ldgt7"
Oct 30 22:07:30.374078 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:07:30.373779    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e17f2f4-e217-43cf-81ba-2bfb2095dbd6-istio-podinfo\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ldgt7\" (UID: \"0e17f2f4-e217-43cf-81ba-2bfb2095dbd6\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ldgt7"
Oct 30 22:07:30.374078 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:07:30.373829    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dr7d\" (UniqueName: \"kubernetes.io/projected/0e17f2f4-e217-43cf-81ba-2bfb2095dbd6-kube-api-access-7dr7d\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ldgt7\" (UID: \"0e17f2f4-e217-43cf-81ba-2bfb2095dbd6\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ldgt7"
Oct 30 22:07:30.374078 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:07:30.373960    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e17f2f4-e217-43cf-81ba-2bfb2095dbd6-credential-socket\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ldgt7\" (UID: \"0e17f2f4-e217-43cf-81ba-2bfb2095dbd6\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ldgt7"
Oct 30 22:07:30.374078 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:07:30.374066    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e17f2f4-e217-43cf-81ba-2bfb2095dbd6-istio-data\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ldgt7\" (UID: \"0e17f2f4-e217-43cf-81ba-2bfb2095dbd6\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ldgt7"
Oct 30 22:07:30.374253 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:07:30.374131    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e17f2f4-e217-43cf-81ba-2bfb2095dbd6-workload-socket\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ldgt7\" (UID: \"0e17f2f4-e217-43cf-81ba-2bfb2095dbd6\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ldgt7"
Oct 30 22:07:30.374356 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:07:30.374337    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e17f2f4-e217-43cf-81ba-2bfb2095dbd6-workload-certs\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ldgt7\" (UID: \"0e17f2f4-e217-43cf-81ba-2bfb2095dbd6\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ldgt7"
Oct 30 22:07:30.374425 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:07:30.374354    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e17f2f4-e217-43cf-81ba-2bfb2095dbd6-istiod-ca-cert\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ldgt7\" (UID: \"0e17f2f4-e217-43cf-81ba-2bfb2095dbd6\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ldgt7"
Oct 30 22:07:30.375982 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:07:30.375959    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e17f2f4-e217-43cf-81ba-2bfb2095dbd6-istio-envoy\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ldgt7\" (UID: \"0e17f2f4-e217-43cf-81ba-2bfb2095dbd6\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ldgt7"
Oct 30 22:07:30.376458 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:07:30.376442    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e17f2f4-e217-43cf-81ba-2bfb2095dbd6-istio-podinfo\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ldgt7\" (UID: \"0e17f2f4-e217-43cf-81ba-2bfb2095dbd6\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ldgt7"
Oct 30 22:07:30.431690 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:07:30.431664    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e17f2f4-e217-43cf-81ba-2bfb2095dbd6-istio-token\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ldgt7\" (UID: \"0e17f2f4-e217-43cf-81ba-2bfb2095dbd6\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ldgt7"
Oct 30 22:07:30.431818 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:07:30.431797    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dr7d\" (UniqueName: \"kubernetes.io/projected/0e17f2f4-e217-43cf-81ba-2bfb2095dbd6-kube-api-access-7dr7d\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ldgt7\" (UID: \"0e17f2f4-e217-43cf-81ba-2bfb2095dbd6\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ldgt7"
Oct 30 22:07:30.521937 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:07:30.521866    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ldgt7"
Oct 30 22:07:30.554888 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:07:30.554857    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-65e59f/premium-w4-65e59f-kserve-6dc49468f8-rlzhx"
Oct 30 22:07:30.963338 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:07:30.963258    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ldgt7"]
Oct 30 22:07:30.968292 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:07:30.968267    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e17f2f4_e217_43cf_81ba_2bfb2095dbd6.slice/crio-bbf89821c56c69ae802e32748e2dae32b3c7ef3c53f6187fb9948aa057138542 WatchSource:0}: Error finding container bbf89821c56c69ae802e32748e2dae32b3c7ef3c53f6187fb9948aa057138542: Status 404 returned error can't find the container with id bbf89821c56c69ae802e32748e2dae32b3c7ef3c53f6187fb9948aa057138542
Oct 30 22:07:30.970747 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:07:30.970718    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:07:30.970814 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:07:30.970785    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:07:30.970854 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:07:30.970815    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:07:31.554266 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:07:31.554224    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ldgt7" event={"ID":"0e17f2f4-e217-43cf-81ba-2bfb2095dbd6","Type":"ContainerStarted","Data":"21e5075edd24d61f4ec91b159a9a0ffa69bdae573523d0dc2eeff217c8a37caf"}
Oct 30 22:07:31.554740 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:07:31.554292    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ldgt7" event={"ID":"0e17f2f4-e217-43cf-81ba-2bfb2095dbd6","Type":"ContainerStarted","Data":"bbf89821c56c69ae802e32748e2dae32b3c7ef3c53f6187fb9948aa057138542"}
Oct 30 22:07:31.670531 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:07:31.670464    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ldgt7" podStartSLOduration=1.6704443279999999 podStartE2EDuration="1.670444328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:07:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:31.669595741 +0000 UTC m=+1029.273903294" watchObservedRunningTime="2026-10-30 22:07:31.670444328 +0000 UTC m=+1029.274751882"
Oct 30 22:07:32.546426 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:07:32.545490    5678 patch_prober.go:29] interesting pod/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ldgt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Oct 30 22:07:32.546426 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:07:32.545564    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ldgt7" podUID="0e17f2f4-e217-43cf-81ba-2bfb2095dbd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Oct 30 22:07:32.546426 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:07:32.546136    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ldgt7"
Oct 30 22:07:33.539424 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:07:33.525671    5678 patch_prober.go:29] interesting pod/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ldgt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Oct 30 22:07:33.539424 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:07:33.525728    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ldgt7" podUID="0e17f2f4-e217-43cf-81ba-2bfb2095dbd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Oct 30 22:07:34.539448 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:07:34.523669    5678 patch_prober.go:29] interesting pod/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ldgt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Oct 30 22:07:34.539448 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:07:34.523724    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ldgt7" podUID="0e17f2f4-e217-43cf-81ba-2bfb2095dbd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Oct 30 22:07:35.547221 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:07:35.547182    5678 patch_prober.go:29] interesting pod/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ldgt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Oct 30 22:07:35.547784 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:07:35.547754    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ldgt7" podUID="0e17f2f4-e217-43cf-81ba-2bfb2095dbd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Oct 30 22:07:36.525752 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:07:36.525703    5678 patch_prober.go:29] interesting pod/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ldgt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Oct 30 22:07:36.526033 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:07:36.525989    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ldgt7" podUID="0e17f2f4-e217-43cf-81ba-2bfb2095dbd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Oct 30 22:07:37.716467 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:07:37.716432    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ldgt7"
Oct 30 22:07:37.717387 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:07:37.717362    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ldgt7"
Oct 30 22:07:37.719604 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:07:37.719581    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ldgt7"
Oct 30 22:07:40.571143 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:07:40.571107    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 22:07:52.785709 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:07:52.785669    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-826abc20-4zllk"]
Oct 30 22:07:52.789027 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:07:52.789003    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-826abc20-4zllk"
Oct 30 22:07:52.844556 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:07:52.844521    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-826abc20-4zllk"]
Oct 30 22:07:52.852177 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:07:52.852146    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vl7pg\" (UniqueName: \"kubernetes.io/projected/938761fa-a709-4b8b-ba91-5caf62caeee2-kube-api-access-vl7pg\") pod \"maas-api-revoke-sub-826abc20-4zllk\" (UID: \"938761fa-a709-4b8b-ba91-5caf62caeee2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-826abc20-4zllk"
Oct 30 22:07:52.852358 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:07:52.852340    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/938761fa-a709-4b8b-ba91-5caf62caeee2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-826abc20-4zllk\" (UID: \"938761fa-a709-4b8b-ba91-5caf62caeee2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-826abc20-4zllk"
Oct 30 22:07:52.852428 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:07:52.852396    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/938761fa-a709-4b8b-ba91-5caf62caeee2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-826abc20-4zllk\" (UID: \"938761fa-a709-4b8b-ba91-5caf62caeee2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-826abc20-4zllk"
Oct 30 22:07:52.887752 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:07:52.887723    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:07:52.887752 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:07:52.887748    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5rvcj\""
Oct 30 22:07:52.953549 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:07:52.953524    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vl7pg\" (UniqueName: \"kubernetes.io/projected/938761fa-a709-4b8b-ba91-5caf62caeee2-kube-api-access-vl7pg\") pod \"maas-api-revoke-sub-826abc20-4zllk\" (UID: \"938761fa-a709-4b8b-ba91-5caf62caeee2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-826abc20-4zllk"
Oct 30 22:07:52.953690 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:07:52.953664    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/938761fa-a709-4b8b-ba91-5caf62caeee2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-826abc20-4zllk\" (UID: \"938761fa-a709-4b8b-ba91-5caf62caeee2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-826abc20-4zllk"
Oct 30 22:07:52.953755 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:07:52.953729    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/938761fa-a709-4b8b-ba91-5caf62caeee2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-826abc20-4zllk\" (UID: \"938761fa-a709-4b8b-ba91-5caf62caeee2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-826abc20-4zllk"
Oct 30 22:07:52.954601 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:07:52.954565    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/938761fa-a709-4b8b-ba91-5caf62caeee2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-826abc20-4zllk\" (UID: \"938761fa-a709-4b8b-ba91-5caf62caeee2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-826abc20-4zllk"
Oct 30 22:07:52.956414 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:07:52.956379    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/938761fa-a709-4b8b-ba91-5caf62caeee2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-826abc20-4zllk\" (UID: \"938761fa-a709-4b8b-ba91-5caf62caeee2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-826abc20-4zllk"
Oct 30 22:07:53.011705 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:07:53.011678    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vl7pg\" (UniqueName: \"kubernetes.io/projected/938761fa-a709-4b8b-ba91-5caf62caeee2-kube-api-access-vl7pg\") pod \"maas-api-revoke-sub-826abc20-4zllk\" (UID: \"938761fa-a709-4b8b-ba91-5caf62caeee2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-826abc20-4zllk"
Oct 30 22:07:53.101735 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:07:53.101670    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-826abc20-4zllk"
Oct 30 22:07:53.483394 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:07:53.483363    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-826abc20-4zllk"]
Oct 30 22:07:53.485685 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:07:53.485656    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod938761fa_a709_4b8b_ba91_5caf62caeee2.slice/crio-4ffd9c587cc985cecf5d7d1b62be4616f7de082151e852e76b40f2ad9db85697 WatchSource:0}: Error finding container 4ffd9c587cc985cecf5d7d1b62be4616f7de082151e852e76b40f2ad9db85697: Status 404 returned error can't find the container with id 4ffd9c587cc985cecf5d7d1b62be4616f7de082151e852e76b40f2ad9db85697
Oct 30 22:07:53.488034 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:07:53.488009    5678 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 22:07:53.685340 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:07:53.685301    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-826abc20-4zllk" event={"ID":"938761fa-a709-4b8b-ba91-5caf62caeee2","Type":"ContainerStarted","Data":"4ffd9c587cc985cecf5d7d1b62be4616f7de082151e852e76b40f2ad9db85697"}
Oct 30 22:07:58.718657 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:07:58.718619    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-826abc20-4zllk" event={"ID":"938761fa-a709-4b8b-ba91-5caf62caeee2","Type":"ContainerStarted","Data":"9983dc645593397b1963b2f6901b553d1c6083868a8550871624f89e29c535af"}
Oct 30 22:07:58.832050 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:07:58.831987    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-826abc20-4zllk" podStartSLOduration=2.4081013479999998 podStartE2EDuration="6.831972362s" totalImagesPullingTime="4.423871014s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:07:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:58.830917871 +0000 UTC m=+1056.435225436" watchObservedRunningTime="2026-10-30 22:07:58.831972362 +0000 UTC m=+1056.436279914"
Oct 30 22:07:59.733641 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:07:59.733603    5678 generic.go:356] "Generic (PLEG): container finished" podID="938761fa-a709-4b8b-ba91-5caf62caeee2" containerID="9983dc645593397b1963b2f6901b553d1c6083868a8550871624f89e29c535af" exitCode=0
Oct 30 22:07:59.734117 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:07:59.733684    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-826abc20-4zllk" event={"ID":"938761fa-a709-4b8b-ba91-5caf62caeee2","Type":"ContainerDied","Data":"9983dc645593397b1963b2f6901b553d1c6083868a8550871624f89e29c535af"}
Oct 30 22:08:00.887758 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:00.887733    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-826abc20-4zllk"
Oct 30 22:08:00.946633 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:00.946599    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/938761fa-a709-4b8b-ba91-5caf62caeee2-kube-api-access-vl7pg\" (UniqueName: \"kubernetes.io/projected/938761fa-a709-4b8b-ba91-5caf62caeee2-kube-api-access-vl7pg\") pod \"938761fa-a709-4b8b-ba91-5caf62caeee2\" (UID: \"938761fa-a709-4b8b-ba91-5caf62caeee2\") "
Oct 30 22:08:00.946814 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:00.946699    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/938761fa-a709-4b8b-ba91-5caf62caeee2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/938761fa-a709-4b8b-ba91-5caf62caeee2-maas-api-openshift-ca\") pod \"938761fa-a709-4b8b-ba91-5caf62caeee2\" (UID: \"938761fa-a709-4b8b-ba91-5caf62caeee2\") "
Oct 30 22:08:00.946880 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:00.946823    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/938761fa-a709-4b8b-ba91-5caf62caeee2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/938761fa-a709-4b8b-ba91-5caf62caeee2-maas-api-xks-ca\") pod \"938761fa-a709-4b8b-ba91-5caf62caeee2\" (UID: \"938761fa-a709-4b8b-ba91-5caf62caeee2\") "
Oct 30 22:08:00.947046 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:00.947021    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/938761fa-a709-4b8b-ba91-5caf62caeee2-maas-api-openshift-ca" pod "938761fa-a709-4b8b-ba91-5caf62caeee2" (UID: "938761fa-a709-4b8b-ba91-5caf62caeee2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:08:00.947195 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:00.947169    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/938761fa-a709-4b8b-ba91-5caf62caeee2-maas-api-openshift-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:00.948839 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:00.948815    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/938761fa-a709-4b8b-ba91-5caf62caeee2-maas-api-xks-ca" pod "938761fa-a709-4b8b-ba91-5caf62caeee2" (UID: "938761fa-a709-4b8b-ba91-5caf62caeee2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:08:00.948934 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:00.948914    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/938761fa-a709-4b8b-ba91-5caf62caeee2-kube-api-access-vl7pg" pod "938761fa-a709-4b8b-ba91-5caf62caeee2" (UID: "938761fa-a709-4b8b-ba91-5caf62caeee2"). InnerVolumeSpecName "kube-api-access-vl7pg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:08:01.048744 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:01.048672    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/938761fa-a709-4b8b-ba91-5caf62caeee2-maas-api-xks-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:01.048744 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:01.048698    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vl7pg\" (UniqueName: \"kubernetes.io/projected/938761fa-a709-4b8b-ba91-5caf62caeee2-kube-api-access-vl7pg\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:01.746459 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:01.746426    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-826abc20-4zllk"
Oct 30 22:08:01.746633 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:01.746433    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-826abc20-4zllk" event={"ID":"938761fa-a709-4b8b-ba91-5caf62caeee2","Type":"ContainerDied","Data":"4ffd9c587cc985cecf5d7d1b62be4616f7de082151e852e76b40f2ad9db85697"}
Oct 30 22:08:01.746633 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:01.746547    5678 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4ffd9c587cc985cecf5d7d1b62be4616f7de082151e852e76b40f2ad9db85697"
Oct 30 22:08:10.571878 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:08:10.571838    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 22:08:24.339667 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:24.339630    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-qjw2b"]
Oct 30 22:08:24.340292 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:24.339877    5678 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-qjw2b" podUID="5d9083c7-76bf-4db9-be88-f8045b5d3c77" containerName="istio-proxy" containerID="cri-o://27f781697fae8b6a60da646656bf76cfa4fb21de2a9deddff449594215ceecdf" gracePeriod=30
Oct 30 22:08:24.740727 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:24.740691    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33af867b-fmbw4"]
Oct 30 22:08:24.741162 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:24.741150    5678 cpu_manager.go:414] "containerMap: removing container" podUID="938761fa-a709-4b8b-ba91-5caf62caeee2" containerName="revoke-keys"
Oct 30 22:08:24.741206 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:24.741163    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="938761fa-a709-4b8b-ba91-5caf62caeee2" containerName="revoke-keys"
Oct 30 22:08:24.741273 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:24.741264    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="938761fa-a709-4b8b-ba91-5caf62caeee2" containerName="revoke-keys"
Oct 30 22:08:24.743943 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:24.743925    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33af867b-fmbw4"
Oct 30 22:08:24.797526 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:24.797484    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:08:24.797526 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:24.797513    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5rvcj\""
Oct 30 22:08:24.800755 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:24.800731    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33af867b-fmbw4"]
Oct 30 22:08:24.847996 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:24.847966    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e093ed7d-278e-45d1-8018-b7db3da90d17-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33af867b-fmbw4\" (UID: \"e093ed7d-278e-45d1-8018-b7db3da90d17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33af867b-fmbw4"
Oct 30 22:08:24.848169 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:24.848024    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bp2v5\" (UniqueName: \"kubernetes.io/projected/e093ed7d-278e-45d1-8018-b7db3da90d17-kube-api-access-bp2v5\") pod \"maas-api-revoke-sub-33af867b-fmbw4\" (UID: \"e093ed7d-278e-45d1-8018-b7db3da90d17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33af867b-fmbw4"
Oct 30 22:08:24.848169 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:24.848138    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e093ed7d-278e-45d1-8018-b7db3da90d17-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33af867b-fmbw4\" (UID: \"e093ed7d-278e-45d1-8018-b7db3da90d17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33af867b-fmbw4"
Oct 30 22:08:24.949363 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:24.949331    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e093ed7d-278e-45d1-8018-b7db3da90d17-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33af867b-fmbw4\" (UID: \"e093ed7d-278e-45d1-8018-b7db3da90d17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33af867b-fmbw4"
Oct 30 22:08:24.949576 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:24.949383    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bp2v5\" (UniqueName: \"kubernetes.io/projected/e093ed7d-278e-45d1-8018-b7db3da90d17-kube-api-access-bp2v5\") pod \"maas-api-revoke-sub-33af867b-fmbw4\" (UID: \"e093ed7d-278e-45d1-8018-b7db3da90d17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33af867b-fmbw4"
Oct 30 22:08:24.949576 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:24.949451    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e093ed7d-278e-45d1-8018-b7db3da90d17-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33af867b-fmbw4\" (UID: \"e093ed7d-278e-45d1-8018-b7db3da90d17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33af867b-fmbw4"
Oct 30 22:08:24.949969 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:24.949950    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e093ed7d-278e-45d1-8018-b7db3da90d17-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33af867b-fmbw4\" (UID: \"e093ed7d-278e-45d1-8018-b7db3da90d17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33af867b-fmbw4"
Oct 30 22:08:24.951803 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:24.951780    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e093ed7d-278e-45d1-8018-b7db3da90d17-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33af867b-fmbw4\" (UID: \"e093ed7d-278e-45d1-8018-b7db3da90d17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33af867b-fmbw4"
Oct 30 22:08:25.008213 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:25.008153    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bp2v5\" (UniqueName: \"kubernetes.io/projected/e093ed7d-278e-45d1-8018-b7db3da90d17-kube-api-access-bp2v5\") pod \"maas-api-revoke-sub-33af867b-fmbw4\" (UID: \"e093ed7d-278e-45d1-8018-b7db3da90d17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33af867b-fmbw4"
Oct 30 22:08:25.056085 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:25.056059    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33af867b-fmbw4"
Oct 30 22:08:25.433628 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:25.433597    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33af867b-fmbw4"]
Oct 30 22:08:25.435605 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:08:25.435577    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode093ed7d_278e_45d1_8018_b7db3da90d17.slice/crio-9a002d502b3d7305b46b0087ec7f3896ad5d8e65c4dbb3438469c5c879686709 WatchSource:0}: Error finding container 9a002d502b3d7305b46b0087ec7f3896ad5d8e65c4dbb3438469c5c879686709: Status 404 returned error can't find the container with id 9a002d502b3d7305b46b0087ec7f3896ad5d8e65c4dbb3438469c5c879686709
Oct 30 22:08:25.905306 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:25.905269    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33af867b-fmbw4" event={"ID":"e093ed7d-278e-45d1-8018-b7db3da90d17","Type":"ContainerStarted","Data":"66940fb240dfab6c3e14322d89fd1052a48ee723a31c2e5d378c08373b370c89"}
Oct 30 22:08:25.905498 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:25.905320    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33af867b-fmbw4" event={"ID":"e093ed7d-278e-45d1-8018-b7db3da90d17","Type":"ContainerStarted","Data":"9a002d502b3d7305b46b0087ec7f3896ad5d8e65c4dbb3438469c5c879686709"}
Oct 30 22:08:26.018061 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:26.017959    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33af867b-fmbw4" podStartSLOduration=2.017944679 podStartE2EDuration="2.017944679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:08:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:08:26.015568964 +0000 UTC m=+1083.619876517" watchObservedRunningTime="2026-10-30 22:08:26.017944679 +0000 UTC m=+1083.622252231"
Oct 30 22:08:26.914273 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:26.914237    5678 generic.go:356] "Generic (PLEG): container finished" podID="e093ed7d-278e-45d1-8018-b7db3da90d17" containerID="66940fb240dfab6c3e14322d89fd1052a48ee723a31c2e5d378c08373b370c89" exitCode=0
Oct 30 22:08:26.914679 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:26.914307    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33af867b-fmbw4" event={"ID":"e093ed7d-278e-45d1-8018-b7db3da90d17","Type":"ContainerDied","Data":"66940fb240dfab6c3e14322d89fd1052a48ee723a31c2e5d378c08373b370c89"}
Oct 30 22:08:28.064900 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:28.064874    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33af867b-fmbw4"
Oct 30 22:08:28.192061 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:28.191985    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e093ed7d-278e-45d1-8018-b7db3da90d17-kube-api-access-bp2v5\" (UniqueName: \"kubernetes.io/projected/e093ed7d-278e-45d1-8018-b7db3da90d17-kube-api-access-bp2v5\") pod \"e093ed7d-278e-45d1-8018-b7db3da90d17\" (UID: \"e093ed7d-278e-45d1-8018-b7db3da90d17\") "
Oct 30 22:08:28.192185 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:28.192110    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e093ed7d-278e-45d1-8018-b7db3da90d17-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e093ed7d-278e-45d1-8018-b7db3da90d17-maas-api-openshift-ca\") pod \"e093ed7d-278e-45d1-8018-b7db3da90d17\" (UID: \"e093ed7d-278e-45d1-8018-b7db3da90d17\") "
Oct 30 22:08:28.192251 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:28.192235    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e093ed7d-278e-45d1-8018-b7db3da90d17-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e093ed7d-278e-45d1-8018-b7db3da90d17-maas-api-xks-ca\") pod \"e093ed7d-278e-45d1-8018-b7db3da90d17\" (UID: \"e093ed7d-278e-45d1-8018-b7db3da90d17\") "
Oct 30 22:08:28.192573 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:28.192543    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e093ed7d-278e-45d1-8018-b7db3da90d17-maas-api-openshift-ca" pod "e093ed7d-278e-45d1-8018-b7db3da90d17" (UID: "e093ed7d-278e-45d1-8018-b7db3da90d17"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:08:28.192689 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:28.192670    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e093ed7d-278e-45d1-8018-b7db3da90d17-maas-api-openshift-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:28.194204 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:28.194177    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e093ed7d-278e-45d1-8018-b7db3da90d17-kube-api-access-bp2v5" pod "e093ed7d-278e-45d1-8018-b7db3da90d17" (UID: "e093ed7d-278e-45d1-8018-b7db3da90d17"). InnerVolumeSpecName "kube-api-access-bp2v5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:08:28.194291 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:28.194266    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e093ed7d-278e-45d1-8018-b7db3da90d17-maas-api-xks-ca" pod "e093ed7d-278e-45d1-8018-b7db3da90d17" (UID: "e093ed7d-278e-45d1-8018-b7db3da90d17"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:08:28.294155 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:28.294122    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bp2v5\" (UniqueName: \"kubernetes.io/projected/e093ed7d-278e-45d1-8018-b7db3da90d17-kube-api-access-bp2v5\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:28.294155 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:28.294150    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e093ed7d-278e-45d1-8018-b7db3da90d17-maas-api-xks-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:28.927335 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:28.927305    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33af867b-fmbw4"
Oct 30 22:08:28.927507 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:28.927331    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33af867b-fmbw4" event={"ID":"e093ed7d-278e-45d1-8018-b7db3da90d17","Type":"ContainerDied","Data":"9a002d502b3d7305b46b0087ec7f3896ad5d8e65c4dbb3438469c5c879686709"}
Oct 30 22:08:28.927507 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:28.927380    5678 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9a002d502b3d7305b46b0087ec7f3896ad5d8e65c4dbb3438469c5c879686709"
Oct 30 22:08:29.598146 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:29.598123    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-qjw2b"
Oct 30 22:08:29.710292 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:29.710263    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d9083c7-76bf-4db9-be88-f8045b5d3c77-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d9083c7-76bf-4db9-be88-f8045b5d3c77-workload-certs\") pod \"5d9083c7-76bf-4db9-be88-f8045b5d3c77\" (UID: \"5d9083c7-76bf-4db9-be88-f8045b5d3c77\") "
Oct 30 22:08:29.710515 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:29.710319    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d9083c7-76bf-4db9-be88-f8045b5d3c77-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d9083c7-76bf-4db9-be88-f8045b5d3c77-istio-data\") pod \"5d9083c7-76bf-4db9-be88-f8045b5d3c77\" (UID: \"5d9083c7-76bf-4db9-be88-f8045b5d3c77\") "
Oct 30 22:08:29.710515 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:29.710341    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d9083c7-76bf-4db9-be88-f8045b5d3c77-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d9083c7-76bf-4db9-be88-f8045b5d3c77-workload-socket\") pod \"5d9083c7-76bf-4db9-be88-f8045b5d3c77\" (UID: \"5d9083c7-76bf-4db9-be88-f8045b5d3c77\") "
Oct 30 22:08:29.710515 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:29.710433    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d9083c7-76bf-4db9-be88-f8045b5d3c77-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d9083c7-76bf-4db9-be88-f8045b5d3c77-istio-envoy\") pod \"5d9083c7-76bf-4db9-be88-f8045b5d3c77\" (UID: \"5d9083c7-76bf-4db9-be88-f8045b5d3c77\") "
Oct 30 22:08:29.710515 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:29.710461    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d9083c7-76bf-4db9-be88-f8045b5d3c77-kube-api-access-tlzd8\" (UniqueName: \"kubernetes.io/projected/5d9083c7-76bf-4db9-be88-f8045b5d3c77-kube-api-access-tlzd8\") pod \"5d9083c7-76bf-4db9-be88-f8045b5d3c77\" (UID: \"5d9083c7-76bf-4db9-be88-f8045b5d3c77\") "
Oct 30 22:08:29.710515 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:29.710480    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d9083c7-76bf-4db9-be88-f8045b5d3c77-istio-token\" (UniqueName: \"kubernetes.io/projected/5d9083c7-76bf-4db9-be88-f8045b5d3c77-istio-token\") pod \"5d9083c7-76bf-4db9-be88-f8045b5d3c77\" (UID: \"5d9083c7-76bf-4db9-be88-f8045b5d3c77\") "
Oct 30 22:08:29.710791 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:29.710575    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d9083c7-76bf-4db9-be88-f8045b5d3c77-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d9083c7-76bf-4db9-be88-f8045b5d3c77-istiod-ca-cert\") pod \"5d9083c7-76bf-4db9-be88-f8045b5d3c77\" (UID: \"5d9083c7-76bf-4db9-be88-f8045b5d3c77\") "
Oct 30 22:08:29.710791 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:29.710627    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d9083c7-76bf-4db9-be88-f8045b5d3c77-istio-data" pod "5d9083c7-76bf-4db9-be88-f8045b5d3c77" (UID: "5d9083c7-76bf-4db9-be88-f8045b5d3c77"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:08:29.710791 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:29.710635    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d9083c7-76bf-4db9-be88-f8045b5d3c77-workload-certs" pod "5d9083c7-76bf-4db9-be88-f8045b5d3c77" (UID: "5d9083c7-76bf-4db9-be88-f8045b5d3c77"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:08:29.710791 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:29.710639    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d9083c7-76bf-4db9-be88-f8045b5d3c77-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d9083c7-76bf-4db9-be88-f8045b5d3c77-credential-socket\") pod \"5d9083c7-76bf-4db9-be88-f8045b5d3c77\" (UID: \"5d9083c7-76bf-4db9-be88-f8045b5d3c77\") "
Oct 30 22:08:29.710791 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:29.710734    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d9083c7-76bf-4db9-be88-f8045b5d3c77-workload-socket" pod "5d9083c7-76bf-4db9-be88-f8045b5d3c77" (UID: "5d9083c7-76bf-4db9-be88-f8045b5d3c77"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:08:29.710791 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:29.710755    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5d9083c7-76bf-4db9-be88-f8045b5d3c77-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d9083c7-76bf-4db9-be88-f8045b5d3c77-istio-podinfo\") pod \"5d9083c7-76bf-4db9-be88-f8045b5d3c77\" (UID: \"5d9083c7-76bf-4db9-be88-f8045b5d3c77\") "
Oct 30 22:08:29.711088 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:29.710876    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d9083c7-76bf-4db9-be88-f8045b5d3c77-credential-socket" pod "5d9083c7-76bf-4db9-be88-f8045b5d3c77" (UID: "5d9083c7-76bf-4db9-be88-f8045b5d3c77"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:08:29.711088 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:29.710952    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d9083c7-76bf-4db9-be88-f8045b5d3c77-istiod-ca-cert" pod "5d9083c7-76bf-4db9-be88-f8045b5d3c77" (UID: "5d9083c7-76bf-4db9-be88-f8045b5d3c77"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:08:29.711276 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:29.711256    5678 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d9083c7-76bf-4db9-be88-f8045b5d3c77-workload-certs\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:29.711332 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:29.711282    5678 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d9083c7-76bf-4db9-be88-f8045b5d3c77-istio-data\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:29.711332 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:29.711300    5678 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d9083c7-76bf-4db9-be88-f8045b5d3c77-workload-socket\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:29.711332 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:29.711315    5678 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d9083c7-76bf-4db9-be88-f8045b5d3c77-istiod-ca-cert\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:29.711332 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:29.711328    5678 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d9083c7-76bf-4db9-be88-f8045b5d3c77-credential-socket\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:29.712984 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:29.712955    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d9083c7-76bf-4db9-be88-f8045b5d3c77-istio-token" pod "5d9083c7-76bf-4db9-be88-f8045b5d3c77" (UID: "5d9083c7-76bf-4db9-be88-f8045b5d3c77"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:08:29.713094 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:29.713028    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5d9083c7-76bf-4db9-be88-f8045b5d3c77-istio-podinfo" pod "5d9083c7-76bf-4db9-be88-f8045b5d3c77" (UID: "5d9083c7-76bf-4db9-be88-f8045b5d3c77"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:08:29.713169 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:29.713140    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d9083c7-76bf-4db9-be88-f8045b5d3c77-kube-api-access-tlzd8" pod "5d9083c7-76bf-4db9-be88-f8045b5d3c77" (UID: "5d9083c7-76bf-4db9-be88-f8045b5d3c77"). InnerVolumeSpecName "kube-api-access-tlzd8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:08:29.713169 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:29.713144    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d9083c7-76bf-4db9-be88-f8045b5d3c77-istio-envoy" pod "5d9083c7-76bf-4db9-be88-f8045b5d3c77" (UID: "5d9083c7-76bf-4db9-be88-f8045b5d3c77"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:08:29.812575 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:29.812545    5678 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d9083c7-76bf-4db9-be88-f8045b5d3c77-istio-envoy\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:29.812575 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:29.812573    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlzd8\" (UniqueName: \"kubernetes.io/projected/5d9083c7-76bf-4db9-be88-f8045b5d3c77-kube-api-access-tlzd8\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:29.812575 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:29.812584    5678 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d9083c7-76bf-4db9-be88-f8045b5d3c77-istio-token\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:29.812803 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:29.812593    5678 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d9083c7-76bf-4db9-be88-f8045b5d3c77-istio-podinfo\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:29.934668 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:29.934636    5678 generic.go:356] "Generic (PLEG): container finished" podID="5d9083c7-76bf-4db9-be88-f8045b5d3c77" containerID="27f781697fae8b6a60da646656bf76cfa4fb21de2a9deddff449594215ceecdf" exitCode=0
Oct 30 22:08:29.934819 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:29.934696    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-qjw2b"
Oct 30 22:08:29.934819 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:29.934715    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-qjw2b" event={"ID":"5d9083c7-76bf-4db9-be88-f8045b5d3c77","Type":"ContainerDied","Data":"27f781697fae8b6a60da646656bf76cfa4fb21de2a9deddff449594215ceecdf"}
Oct 30 22:08:29.934819 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:29.934777    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-qjw2b" event={"ID":"5d9083c7-76bf-4db9-be88-f8045b5d3c77","Type":"ContainerDied","Data":"dc1aa9c0b849afea337d4aaf0a1496b5c8ba58324dc949790cec1c51149b3136"}
Oct 30 22:08:29.934819 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:29.934804    5678 scope.go:122] "RemoveContainer" containerID="27f781697fae8b6a60da646656bf76cfa4fb21de2a9deddff449594215ceecdf"
Oct 30 22:08:29.945842 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:29.945824    5678 scope.go:122] "RemoveContainer" containerID="27f781697fae8b6a60da646656bf76cfa4fb21de2a9deddff449594215ceecdf"
Oct 30 22:08:29.946165 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:08:29.946143    5678 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27f781697fae8b6a60da646656bf76cfa4fb21de2a9deddff449594215ceecdf\": container with ID starting with 27f781697fae8b6a60da646656bf76cfa4fb21de2a9deddff449594215ceecdf not found: ID does not exist" containerID="27f781697fae8b6a60da646656bf76cfa4fb21de2a9deddff449594215ceecdf"
Oct 30 22:08:29.946239 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:29.946176    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27f781697fae8b6a60da646656bf76cfa4fb21de2a9deddff449594215ceecdf"} err="failed to get container status \"27f781697fae8b6a60da646656bf76cfa4fb21de2a9deddff449594215ceecdf\": rpc error: code = NotFound desc = could not find container \"27f781697fae8b6a60da646656bf76cfa4fb21de2a9deddff449594215ceecdf\": container with ID starting with 27f781697fae8b6a60da646656bf76cfa4fb21de2a9deddff449594215ceecdf not found: ID does not exist"
Oct 30 22:08:30.092646 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:30.092562    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-qjw2b"]
Oct 30 22:08:30.099693 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:30.099664    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-qjw2b"]
Oct 30 22:08:30.523075 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:30.523041    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d9083c7-76bf-4db9-be88-f8045b5d3c77" path="/var/lib/kubelet/pods/5d9083c7-76bf-4db9-be88-f8045b5d3c77/volumes"
Oct 30 22:08:31.439177 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:31.439105    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-7jhn9"]
Oct 30 22:08:31.439619 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:31.439605    5678 cpu_manager.go:414] "containerMap: removing container" podUID="5d9083c7-76bf-4db9-be88-f8045b5d3c77" containerName="istio-proxy"
Oct 30 22:08:31.439662 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:31.439620    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d9083c7-76bf-4db9-be88-f8045b5d3c77" containerName="istio-proxy"
Oct 30 22:08:31.439662 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:31.439635    5678 cpu_manager.go:414] "containerMap: removing container" podUID="e093ed7d-278e-45d1-8018-b7db3da90d17" containerName="revoke-keys"
Oct 30 22:08:31.439662 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:31.439642    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e093ed7d-278e-45d1-8018-b7db3da90d17" containerName="revoke-keys"
Oct 30 22:08:31.439755 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:31.439749    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d9083c7-76bf-4db9-be88-f8045b5d3c77" containerName="istio-proxy"
Oct 30 22:08:31.439788 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:31.439758    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="e093ed7d-278e-45d1-8018-b7db3da90d17" containerName="revoke-keys"
Oct 30 22:08:31.447300 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:31.447279    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-7jhn9"
Oct 30 22:08:31.531409 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:31.531359    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fca4f4f-8d98-4ecf-8a41-828fef1ceb56-credential-socket\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-7jhn9\" (UID: \"7fca4f4f-8d98-4ecf-8a41-828fef1ceb56\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-7jhn9"
Oct 30 22:08:31.531409 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:31.531396    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8b9td\" (UniqueName: \"kubernetes.io/projected/7fca4f4f-8d98-4ecf-8a41-828fef1ceb56-kube-api-access-8b9td\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-7jhn9\" (UID: \"7fca4f4f-8d98-4ecf-8a41-828fef1ceb56\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-7jhn9"
Oct 30 22:08:31.531661 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:31.531608    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fca4f4f-8d98-4ecf-8a41-828fef1ceb56-workload-certs\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-7jhn9\" (UID: \"7fca4f4f-8d98-4ecf-8a41-828fef1ceb56\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-7jhn9"
Oct 30 22:08:31.531726 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:31.531658    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fca4f4f-8d98-4ecf-8a41-828fef1ceb56-istio-podinfo\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-7jhn9\" (UID: \"7fca4f4f-8d98-4ecf-8a41-828fef1ceb56\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-7jhn9"
Oct 30 22:08:31.531784 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:31.531746    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fca4f4f-8d98-4ecf-8a41-828fef1ceb56-workload-socket\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-7jhn9\" (UID: \"7fca4f4f-8d98-4ecf-8a41-828fef1ceb56\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-7jhn9"
Oct 30 22:08:31.531784 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:31.531771    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fca4f4f-8d98-4ecf-8a41-828fef1ceb56-istio-data\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-7jhn9\" (UID: \"7fca4f4f-8d98-4ecf-8a41-828fef1ceb56\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-7jhn9"
Oct 30 22:08:31.531893 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:31.531791    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fca4f4f-8d98-4ecf-8a41-828fef1ceb56-istiod-ca-cert\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-7jhn9\" (UID: \"7fca4f4f-8d98-4ecf-8a41-828fef1ceb56\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-7jhn9"
Oct 30 22:08:31.531893 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:31.531828    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fca4f4f-8d98-4ecf-8a41-828fef1ceb56-istio-envoy\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-7jhn9\" (UID: \"7fca4f4f-8d98-4ecf-8a41-828fef1ceb56\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-7jhn9"
Oct 30 22:08:31.532000 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:31.531904    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fca4f4f-8d98-4ecf-8a41-828fef1ceb56-istio-token\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-7jhn9\" (UID: \"7fca4f4f-8d98-4ecf-8a41-828fef1ceb56\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-7jhn9"
Oct 30 22:08:31.545276 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:31.545076    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-65e59f-openshift-default-dockercfg-rvg5q\""
Oct 30 22:08:31.551806 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:31.551779    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-7jhn9"]
Oct 30 22:08:31.633418 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:31.633372    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fca4f4f-8d98-4ecf-8a41-828fef1ceb56-istio-token\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-7jhn9\" (UID: \"7fca4f4f-8d98-4ecf-8a41-828fef1ceb56\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-7jhn9"
Oct 30 22:08:31.633618 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:31.633460    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fca4f4f-8d98-4ecf-8a41-828fef1ceb56-credential-socket\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-7jhn9\" (UID: \"7fca4f4f-8d98-4ecf-8a41-828fef1ceb56\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-7jhn9"
Oct 30 22:08:31.633618 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:31.633487    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8b9td\" (UniqueName: \"kubernetes.io/projected/7fca4f4f-8d98-4ecf-8a41-828fef1ceb56-kube-api-access-8b9td\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-7jhn9\" (UID: \"7fca4f4f-8d98-4ecf-8a41-828fef1ceb56\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-7jhn9"
Oct 30 22:08:31.633957 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:31.633927    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fca4f4f-8d98-4ecf-8a41-828fef1ceb56-workload-certs\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-7jhn9\" (UID: \"7fca4f4f-8d98-4ecf-8a41-828fef1ceb56\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-7jhn9"
Oct 30 22:08:31.634033 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:31.633986    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fca4f4f-8d98-4ecf-8a41-828fef1ceb56-istio-podinfo\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-7jhn9\" (UID: \"7fca4f4f-8d98-4ecf-8a41-828fef1ceb56\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-7jhn9"
Oct 30 22:08:31.634033 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:31.634017    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fca4f4f-8d98-4ecf-8a41-828fef1ceb56-workload-socket\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-7jhn9\" (UID: \"7fca4f4f-8d98-4ecf-8a41-828fef1ceb56\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-7jhn9"
Oct 30 22:08:31.634138 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:31.634056    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fca4f4f-8d98-4ecf-8a41-828fef1ceb56-credential-socket\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-7jhn9\" (UID: \"7fca4f4f-8d98-4ecf-8a41-828fef1ceb56\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-7jhn9"
Oct 30 22:08:31.634138 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:31.634060    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fca4f4f-8d98-4ecf-8a41-828fef1ceb56-istio-data\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-7jhn9\" (UID: \"7fca4f4f-8d98-4ecf-8a41-828fef1ceb56\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-7jhn9"
Oct 30 22:08:31.634138 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:31.634103    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fca4f4f-8d98-4ecf-8a41-828fef1ceb56-istiod-ca-cert\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-7jhn9\" (UID: \"7fca4f4f-8d98-4ecf-8a41-828fef1ceb56\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-7jhn9"
Oct 30 22:08:31.634357 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:31.634140    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fca4f4f-8d98-4ecf-8a41-828fef1ceb56-istio-envoy\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-7jhn9\" (UID: \"7fca4f4f-8d98-4ecf-8a41-828fef1ceb56\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-7jhn9"
Oct 30 22:08:31.634357 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:31.634266    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fca4f4f-8d98-4ecf-8a41-828fef1ceb56-workload-certs\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-7jhn9\" (UID: \"7fca4f4f-8d98-4ecf-8a41-828fef1ceb56\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-7jhn9"
Oct 30 22:08:31.634503 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:31.634342    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fca4f4f-8d98-4ecf-8a41-828fef1ceb56-istio-data\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-7jhn9\" (UID: \"7fca4f4f-8d98-4ecf-8a41-828fef1ceb56\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-7jhn9"
Oct 30 22:08:31.634503 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:31.634387    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fca4f4f-8d98-4ecf-8a41-828fef1ceb56-workload-socket\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-7jhn9\" (UID: \"7fca4f4f-8d98-4ecf-8a41-828fef1ceb56\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-7jhn9"
Oct 30 22:08:31.634939 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:31.634904    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fca4f4f-8d98-4ecf-8a41-828fef1ceb56-istiod-ca-cert\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-7jhn9\" (UID: \"7fca4f4f-8d98-4ecf-8a41-828fef1ceb56\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-7jhn9"
Oct 30 22:08:31.636571 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:31.636543    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fca4f4f-8d98-4ecf-8a41-828fef1ceb56-istio-envoy\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-7jhn9\" (UID: \"7fca4f4f-8d98-4ecf-8a41-828fef1ceb56\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-7jhn9"
Oct 30 22:08:31.636732 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:31.636710    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fca4f4f-8d98-4ecf-8a41-828fef1ceb56-istio-podinfo\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-7jhn9\" (UID: \"7fca4f4f-8d98-4ecf-8a41-828fef1ceb56\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-7jhn9"
Oct 30 22:08:31.692388 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:31.692305    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fca4f4f-8d98-4ecf-8a41-828fef1ceb56-istio-token\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-7jhn9\" (UID: \"7fca4f4f-8d98-4ecf-8a41-828fef1ceb56\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-7jhn9"
Oct 30 22:08:31.692569 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:31.692436    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8b9td\" (UniqueName: \"kubernetes.io/projected/7fca4f4f-8d98-4ecf-8a41-828fef1ceb56-kube-api-access-8b9td\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-7jhn9\" (UID: \"7fca4f4f-8d98-4ecf-8a41-828fef1ceb56\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-7jhn9"
Oct 30 22:08:31.762844 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:31.762805    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-7jhn9"
Oct 30 22:08:32.211929 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:08:32.211845    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fca4f4f_8d98_4ecf_8a41_828fef1ceb56.slice/crio-92642760897d363b993768c54972338be7a2f94e3eefb39a6afad09357c977ea WatchSource:0}: Error finding container 92642760897d363b993768c54972338be7a2f94e3eefb39a6afad09357c977ea: Status 404 returned error can't find the container with id 92642760897d363b993768c54972338be7a2f94e3eefb39a6afad09357c977ea
Oct 30 22:08:32.212256 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:32.212228    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-7jhn9"]
Oct 30 22:08:32.214451 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:32.214388    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:08:32.214557 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:32.214496    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:08:32.214557 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:32.214541    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:08:32.995426 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:32.994972    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-7jhn9" event={"ID":"7fca4f4f-8d98-4ecf-8a41-828fef1ceb56","Type":"ContainerStarted","Data":"6a7fe0598e9b782a421c6b55f3aeee30480aa96ffeb88b9005eb5d317350a8d9"}
Oct 30 22:08:32.995426 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:32.995043    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-7jhn9" event={"ID":"7fca4f4f-8d98-4ecf-8a41-828fef1ceb56","Type":"ContainerStarted","Data":"92642760897d363b993768c54972338be7a2f94e3eefb39a6afad09357c977ea"}
Oct 30 22:08:33.182425 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:33.181415    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-7jhn9" podStartSLOduration=2.181377736 podStartE2EDuration="2.181377736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:08:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:08:33.159840671 +0000 UTC m=+1090.764148226" watchObservedRunningTime="2026-10-30 22:08:33.181377736 +0000 UTC m=+1090.785685289"
Oct 30 22:08:33.763597 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:33.763552    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-7jhn9"
Oct 30 22:08:33.765690 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:33.765663    5678 patch_prober.go:29] interesting pod/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-7jhn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Oct 30 22:08:33.765995 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:33.765950    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-7jhn9" podUID="7fca4f4f-8d98-4ecf-8a41-828fef1ceb56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Oct 30 22:08:34.763920 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:34.763879    5678 patch_prober.go:29] interesting pod/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-7jhn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Oct 30 22:08:34.764298 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:34.763943    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-7jhn9" podUID="7fca4f4f-8d98-4ecf-8a41-828fef1ceb56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Oct 30 22:08:35.094914 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:35.094841    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88b75b32-w7mh9"]
Oct 30 22:08:35.098363 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:35.098343    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88b75b32-w7mh9"
Oct 30 22:08:35.148703 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:35.148676    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:08:35.195453 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:35.195392    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5rvcj\""
Oct 30 22:08:35.198368 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:35.198343    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88b75b32-w7mh9"]
Oct 30 22:08:35.206947 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:35.206920    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc690a1c-e106-4abd-b6ce-0ef6841374cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88b75b32-w7mh9\" (UID: \"bc690a1c-e106-4abd-b6ce-0ef6841374cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88b75b32-w7mh9"
Oct 30 22:08:35.207089 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:35.207037    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc690a1c-e106-4abd-b6ce-0ef6841374cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88b75b32-w7mh9\" (UID: \"bc690a1c-e106-4abd-b6ce-0ef6841374cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88b75b32-w7mh9"
Oct 30 22:08:35.207151 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:35.207092    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7ljl\" (UniqueName: \"kubernetes.io/projected/bc690a1c-e106-4abd-b6ce-0ef6841374cc-kube-api-access-d7ljl\") pod \"maas-api-revoke-sub-88b75b32-w7mh9\" (UID: \"bc690a1c-e106-4abd-b6ce-0ef6841374cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88b75b32-w7mh9"
Oct 30 22:08:35.307988 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:35.307954    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc690a1c-e106-4abd-b6ce-0ef6841374cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88b75b32-w7mh9\" (UID: \"bc690a1c-e106-4abd-b6ce-0ef6841374cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88b75b32-w7mh9"
Oct 30 22:08:35.308179 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:35.308007    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7ljl\" (UniqueName: \"kubernetes.io/projected/bc690a1c-e106-4abd-b6ce-0ef6841374cc-kube-api-access-d7ljl\") pod \"maas-api-revoke-sub-88b75b32-w7mh9\" (UID: \"bc690a1c-e106-4abd-b6ce-0ef6841374cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88b75b32-w7mh9"
Oct 30 22:08:35.308179 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:35.308091    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc690a1c-e106-4abd-b6ce-0ef6841374cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88b75b32-w7mh9\" (UID: \"bc690a1c-e106-4abd-b6ce-0ef6841374cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88b75b32-w7mh9"
Oct 30 22:08:35.308709 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:35.308687    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc690a1c-e106-4abd-b6ce-0ef6841374cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88b75b32-w7mh9\" (UID: \"bc690a1c-e106-4abd-b6ce-0ef6841374cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88b75b32-w7mh9"
Oct 30 22:08:35.310187 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:35.310167    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc690a1c-e106-4abd-b6ce-0ef6841374cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88b75b32-w7mh9\" (UID: \"bc690a1c-e106-4abd-b6ce-0ef6841374cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88b75b32-w7mh9"
Oct 30 22:08:35.367200 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:35.367164    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7ljl\" (UniqueName: \"kubernetes.io/projected/bc690a1c-e106-4abd-b6ce-0ef6841374cc-kube-api-access-d7ljl\") pod \"maas-api-revoke-sub-88b75b32-w7mh9\" (UID: \"bc690a1c-e106-4abd-b6ce-0ef6841374cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88b75b32-w7mh9"
Oct 30 22:08:35.411004 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:35.410974    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88b75b32-w7mh9"
Oct 30 22:08:35.764445 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:35.764339    5678 patch_prober.go:29] interesting pod/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-7jhn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Oct 30 22:08:35.764445 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:35.764391    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-7jhn9" podUID="7fca4f4f-8d98-4ecf-8a41-828fef1ceb56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Oct 30 22:08:35.815812 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:35.815778    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88b75b32-w7mh9"]
Oct 30 22:08:35.816583 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:08:35.816556    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc690a1c_e106_4abd_b6ce_0ef6841374cc.slice/crio-b68c2c3e5d1cf6325e42c62f3bbef545345410c7d56fd3959bb8e3e815aeab61 WatchSource:0}: Error finding container b68c2c3e5d1cf6325e42c62f3bbef545345410c7d56fd3959bb8e3e815aeab61: Status 404 returned error can't find the container with id b68c2c3e5d1cf6325e42c62f3bbef545345410c7d56fd3959bb8e3e815aeab61
Oct 30 22:08:36.053498 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:36.053383    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88b75b32-w7mh9" event={"ID":"bc690a1c-e106-4abd-b6ce-0ef6841374cc","Type":"ContainerStarted","Data":"b68c2c3e5d1cf6325e42c62f3bbef545345410c7d56fd3959bb8e3e815aeab61"}
Oct 30 22:08:36.763782 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:36.763752    5678 patch_prober.go:29] interesting pod/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-7jhn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Oct 30 22:08:36.763957 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:36.763803    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-7jhn9" podUID="7fca4f4f-8d98-4ecf-8a41-828fef1ceb56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Oct 30 22:08:37.061321 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:37.061231    5678 generic.go:356] "Generic (PLEG): container finished" podID="bc690a1c-e106-4abd-b6ce-0ef6841374cc" containerID="4d585125f281551cc003d2da6ef58346e1549a8697fd5835fddd501bf404e870" exitCode=0
Oct 30 22:08:37.061321 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:37.061280    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88b75b32-w7mh9" event={"ID":"bc690a1c-e106-4abd-b6ce-0ef6841374cc","Type":"ContainerDied","Data":"4d585125f281551cc003d2da6ef58346e1549a8697fd5835fddd501bf404e870"}
Oct 30 22:08:37.764090 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:37.764052    5678 patch_prober.go:29] interesting pod/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-7jhn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Oct 30 22:08:37.764270 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:37.764104    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-7jhn9" podUID="7fca4f4f-8d98-4ecf-8a41-828fef1ceb56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Oct 30 22:08:38.248322 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:38.248291    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88b75b32-w7mh9"
Oct 30 22:08:38.367549 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:38.367508    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc690a1c-e106-4abd-b6ce-0ef6841374cc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc690a1c-e106-4abd-b6ce-0ef6841374cc-maas-api-xks-ca\") pod \"bc690a1c-e106-4abd-b6ce-0ef6841374cc\" (UID: \"bc690a1c-e106-4abd-b6ce-0ef6841374cc\") "
Oct 30 22:08:38.367742 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:38.367571    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc690a1c-e106-4abd-b6ce-0ef6841374cc-kube-api-access-d7ljl\" (UniqueName: \"kubernetes.io/projected/bc690a1c-e106-4abd-b6ce-0ef6841374cc-kube-api-access-d7ljl\") pod \"bc690a1c-e106-4abd-b6ce-0ef6841374cc\" (UID: \"bc690a1c-e106-4abd-b6ce-0ef6841374cc\") "
Oct 30 22:08:38.367742 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:38.367660    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc690a1c-e106-4abd-b6ce-0ef6841374cc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc690a1c-e106-4abd-b6ce-0ef6841374cc-maas-api-openshift-ca\") pod \"bc690a1c-e106-4abd-b6ce-0ef6841374cc\" (UID: \"bc690a1c-e106-4abd-b6ce-0ef6841374cc\") "
Oct 30 22:08:38.368002 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:38.367974    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc690a1c-e106-4abd-b6ce-0ef6841374cc-maas-api-openshift-ca" pod "bc690a1c-e106-4abd-b6ce-0ef6841374cc" (UID: "bc690a1c-e106-4abd-b6ce-0ef6841374cc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:08:38.368177 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:38.368157    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc690a1c-e106-4abd-b6ce-0ef6841374cc-maas-api-openshift-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:38.371035 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:38.371004    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc690a1c-e106-4abd-b6ce-0ef6841374cc-kube-api-access-d7ljl" pod "bc690a1c-e106-4abd-b6ce-0ef6841374cc" (UID: "bc690a1c-e106-4abd-b6ce-0ef6841374cc"). InnerVolumeSpecName "kube-api-access-d7ljl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:08:38.371225 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:38.371203    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc690a1c-e106-4abd-b6ce-0ef6841374cc-maas-api-xks-ca" pod "bc690a1c-e106-4abd-b6ce-0ef6841374cc" (UID: "bc690a1c-e106-4abd-b6ce-0ef6841374cc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:08:38.469261 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:38.469226    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7ljl\" (UniqueName: \"kubernetes.io/projected/bc690a1c-e106-4abd-b6ce-0ef6841374cc-kube-api-access-d7ljl\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:38.469261 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:38.469255    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc690a1c-e106-4abd-b6ce-0ef6841374cc-maas-api-xks-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:38.763336 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:38.763256    5678 patch_prober.go:29] interesting pod/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-7jhn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Oct 30 22:08:38.763336 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:38.763310    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-7jhn9" podUID="7fca4f4f-8d98-4ecf-8a41-828fef1ceb56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Oct 30 22:08:39.078674 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:39.078558    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88b75b32-w7mh9"
Oct 30 22:08:39.078674 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:39.078624    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88b75b32-w7mh9" event={"ID":"bc690a1c-e106-4abd-b6ce-0ef6841374cc","Type":"ContainerDied","Data":"b68c2c3e5d1cf6325e42c62f3bbef545345410c7d56fd3959bb8e3e815aeab61"}
Oct 30 22:08:39.078911 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:39.078693    5678 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b68c2c3e5d1cf6325e42c62f3bbef545345410c7d56fd3959bb8e3e815aeab61"
Oct 30 22:08:39.559078 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:39.559045    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d5eb2f4-26gc5"]
Oct 30 22:08:39.559583 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:39.559567    5678 cpu_manager.go:414] "containerMap: removing container" podUID="bc690a1c-e106-4abd-b6ce-0ef6841374cc" containerName="revoke-keys"
Oct 30 22:08:39.559662 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:39.559585    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc690a1c-e106-4abd-b6ce-0ef6841374cc" containerName="revoke-keys"
Oct 30 22:08:39.559716 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:39.559679    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc690a1c-e106-4abd-b6ce-0ef6841374cc" containerName="revoke-keys"
Oct 30 22:08:39.562311 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:39.562291    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d5eb2f4-26gc5"
Oct 30 22:08:39.580329 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:39.580307    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8glvx\" (UniqueName: \"kubernetes.io/projected/8394754e-3f5a-43e1-99df-1349985623f5-kube-api-access-8glvx\") pod \"maas-api-revoke-sub-6d5eb2f4-26gc5\" (UID: \"8394754e-3f5a-43e1-99df-1349985623f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d5eb2f4-26gc5"
Oct 30 22:08:39.580448 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:39.580349    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8394754e-3f5a-43e1-99df-1349985623f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d5eb2f4-26gc5\" (UID: \"8394754e-3f5a-43e1-99df-1349985623f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d5eb2f4-26gc5"
Oct 30 22:08:39.580508 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:39.580488    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8394754e-3f5a-43e1-99df-1349985623f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d5eb2f4-26gc5\" (UID: \"8394754e-3f5a-43e1-99df-1349985623f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d5eb2f4-26gc5"
Oct 30 22:08:39.612158 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:39.612134    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5rvcj\""
Oct 30 22:08:39.614841 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:39.614808    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:08:39.618491 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:39.618469    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d5eb2f4-26gc5"]
Oct 30 22:08:39.681111 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:39.681080    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8394754e-3f5a-43e1-99df-1349985623f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d5eb2f4-26gc5\" (UID: \"8394754e-3f5a-43e1-99df-1349985623f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d5eb2f4-26gc5"
Oct 30 22:08:39.681272 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:39.681175    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8glvx\" (UniqueName: \"kubernetes.io/projected/8394754e-3f5a-43e1-99df-1349985623f5-kube-api-access-8glvx\") pod \"maas-api-revoke-sub-6d5eb2f4-26gc5\" (UID: \"8394754e-3f5a-43e1-99df-1349985623f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d5eb2f4-26gc5"
Oct 30 22:08:39.681272 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:39.681209    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8394754e-3f5a-43e1-99df-1349985623f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d5eb2f4-26gc5\" (UID: \"8394754e-3f5a-43e1-99df-1349985623f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d5eb2f4-26gc5"
Oct 30 22:08:39.681829 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:39.681803    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8394754e-3f5a-43e1-99df-1349985623f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d5eb2f4-26gc5\" (UID: \"8394754e-3f5a-43e1-99df-1349985623f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d5eb2f4-26gc5"
Oct 30 22:08:39.683513 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:39.683494    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8394754e-3f5a-43e1-99df-1349985623f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d5eb2f4-26gc5\" (UID: \"8394754e-3f5a-43e1-99df-1349985623f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d5eb2f4-26gc5"
Oct 30 22:08:39.742847 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:39.742818    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8glvx\" (UniqueName: \"kubernetes.io/projected/8394754e-3f5a-43e1-99df-1349985623f5-kube-api-access-8glvx\") pod \"maas-api-revoke-sub-6d5eb2f4-26gc5\" (UID: \"8394754e-3f5a-43e1-99df-1349985623f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d5eb2f4-26gc5"
Oct 30 22:08:39.767966 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:39.767938    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-7jhn9"
Oct 30 22:08:39.768279 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:39.768261    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-7jhn9"
Oct 30 22:08:39.768971 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:39.768954    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-7jhn9"
Oct 30 22:08:39.874016 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:39.873990    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d5eb2f4-26gc5"
Oct 30 22:08:40.259256 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:40.259227    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d5eb2f4-26gc5"]
Oct 30 22:08:40.260211 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:08:40.260171    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8394754e_3f5a_43e1_99df_1349985623f5.slice/crio-4a0e86fa1c7ac0549a4937872a2d5ce9d7683d045ada2991940cc559934395b6 WatchSource:0}: Error finding container 4a0e86fa1c7ac0549a4937872a2d5ce9d7683d045ada2991940cc559934395b6: Status 404 returned error can't find the container with id 4a0e86fa1c7ac0549a4937872a2d5ce9d7683d045ada2991940cc559934395b6
Oct 30 22:08:40.571372 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:08:40.571339    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 22:08:41.099682 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:41.099647    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d5eb2f4-26gc5" event={"ID":"8394754e-3f5a-43e1-99df-1349985623f5","Type":"ContainerStarted","Data":"4f4c41fd74ea0acea98a40bd9e2a4a127a511bc6317aafac7cd1dd3381a7f493"}
Oct 30 22:08:41.099873 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:41.099701    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d5eb2f4-26gc5" event={"ID":"8394754e-3f5a-43e1-99df-1349985623f5","Type":"ContainerStarted","Data":"4a0e86fa1c7ac0549a4937872a2d5ce9d7683d045ada2991940cc559934395b6"}
Oct 30 22:08:41.212098 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:41.212047    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d5eb2f4-26gc5" podStartSLOduration=2.21203292 podStartE2EDuration="2.21203292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:08:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:08:41.209572114 +0000 UTC m=+1098.813879667" watchObservedRunningTime="2026-10-30 22:08:41.21203292 +0000 UTC m=+1098.816340474"
Oct 30 22:08:42.107559 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:42.107526    5678 generic.go:356] "Generic (PLEG): container finished" podID="8394754e-3f5a-43e1-99df-1349985623f5" containerID="4f4c41fd74ea0acea98a40bd9e2a4a127a511bc6317aafac7cd1dd3381a7f493" exitCode=0
Oct 30 22:08:42.108006 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:42.107610    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d5eb2f4-26gc5" event={"ID":"8394754e-3f5a-43e1-99df-1349985623f5","Type":"ContainerDied","Data":"4f4c41fd74ea0acea98a40bd9e2a4a127a511bc6317aafac7cd1dd3381a7f493"}
Oct 30 22:08:43.249791 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:43.249762    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d5eb2f4-26gc5"
Oct 30 22:08:43.322891 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:43.322861    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8394754e-3f5a-43e1-99df-1349985623f5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8394754e-3f5a-43e1-99df-1349985623f5-maas-api-xks-ca\") pod \"8394754e-3f5a-43e1-99df-1349985623f5\" (UID: \"8394754e-3f5a-43e1-99df-1349985623f5\") "
Oct 30 22:08:43.323050 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:43.322898    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8394754e-3f5a-43e1-99df-1349985623f5-kube-api-access-8glvx\" (UniqueName: \"kubernetes.io/projected/8394754e-3f5a-43e1-99df-1349985623f5-kube-api-access-8glvx\") pod \"8394754e-3f5a-43e1-99df-1349985623f5\" (UID: \"8394754e-3f5a-43e1-99df-1349985623f5\") "
Oct 30 22:08:43.323050 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:43.322952    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8394754e-3f5a-43e1-99df-1349985623f5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8394754e-3f5a-43e1-99df-1349985623f5-maas-api-openshift-ca\") pod \"8394754e-3f5a-43e1-99df-1349985623f5\" (UID: \"8394754e-3f5a-43e1-99df-1349985623f5\") "
Oct 30 22:08:43.323419 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:43.323370    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8394754e-3f5a-43e1-99df-1349985623f5-maas-api-openshift-ca" pod "8394754e-3f5a-43e1-99df-1349985623f5" (UID: "8394754e-3f5a-43e1-99df-1349985623f5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:08:43.324984 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:43.324964    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8394754e-3f5a-43e1-99df-1349985623f5-kube-api-access-8glvx" pod "8394754e-3f5a-43e1-99df-1349985623f5" (UID: "8394754e-3f5a-43e1-99df-1349985623f5"). InnerVolumeSpecName "kube-api-access-8glvx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:08:43.325072 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:43.325051    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8394754e-3f5a-43e1-99df-1349985623f5-maas-api-xks-ca" pod "8394754e-3f5a-43e1-99df-1349985623f5" (UID: "8394754e-3f5a-43e1-99df-1349985623f5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:08:43.424940 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:43.424911    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8394754e-3f5a-43e1-99df-1349985623f5-maas-api-openshift-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:43.424940 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:43.424938    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8394754e-3f5a-43e1-99df-1349985623f5-maas-api-xks-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:43.424940 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:43.424948    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8glvx\" (UniqueName: \"kubernetes.io/projected/8394754e-3f5a-43e1-99df-1349985623f5-kube-api-access-8glvx\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:44.121719 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:44.121687    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d5eb2f4-26gc5"
Oct 30 22:08:44.121867 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:44.121719    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d5eb2f4-26gc5" event={"ID":"8394754e-3f5a-43e1-99df-1349985623f5","Type":"ContainerDied","Data":"4a0e86fa1c7ac0549a4937872a2d5ce9d7683d045ada2991940cc559934395b6"}
Oct 30 22:08:44.121867 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:44.121789    5678 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4a0e86fa1c7ac0549a4937872a2d5ce9d7683d045ada2991940cc559934395b6"
Oct 30 22:08:44.544319 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:44.544238    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74aa8537-zxf2j"]
Oct 30 22:08:44.544963 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:44.544946    5678 cpu_manager.go:414] "containerMap: removing container" podUID="8394754e-3f5a-43e1-99df-1349985623f5" containerName="revoke-keys"
Oct 30 22:08:44.545058 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:44.544965    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8394754e-3f5a-43e1-99df-1349985623f5" containerName="revoke-keys"
Oct 30 22:08:44.545100 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:44.545078    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="8394754e-3f5a-43e1-99df-1349985623f5" containerName="revoke-keys"
Oct 30 22:08:44.548095 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:44.548074    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74aa8537-zxf2j"
Oct 30 22:08:44.637452 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:44.637411    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fm69d\" (UniqueName: \"kubernetes.io/projected/a617d026-0fd3-469b-87ec-47b0f4d85ad5-kube-api-access-fm69d\") pod \"maas-api-revoke-sub-74aa8537-zxf2j\" (UID: \"a617d026-0fd3-469b-87ec-47b0f4d85ad5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74aa8537-zxf2j"
Oct 30 22:08:44.637616 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:44.637572    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a617d026-0fd3-469b-87ec-47b0f4d85ad5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-74aa8537-zxf2j\" (UID: \"a617d026-0fd3-469b-87ec-47b0f4d85ad5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74aa8537-zxf2j"
Oct 30 22:08:44.637775 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:44.637745    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a617d026-0fd3-469b-87ec-47b0f4d85ad5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-74aa8537-zxf2j\" (UID: \"a617d026-0fd3-469b-87ec-47b0f4d85ad5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74aa8537-zxf2j"
Oct 30 22:08:44.644954 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:44.644923    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:08:44.645080 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:44.644979    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5rvcj\""
Oct 30 22:08:44.647924 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:44.647903    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74aa8537-zxf2j"]
Oct 30 22:08:44.739443 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:44.739392    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a617d026-0fd3-469b-87ec-47b0f4d85ad5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-74aa8537-zxf2j\" (UID: \"a617d026-0fd3-469b-87ec-47b0f4d85ad5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74aa8537-zxf2j"
Oct 30 22:08:44.739560 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:44.739480    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fm69d\" (UniqueName: \"kubernetes.io/projected/a617d026-0fd3-469b-87ec-47b0f4d85ad5-kube-api-access-fm69d\") pod \"maas-api-revoke-sub-74aa8537-zxf2j\" (UID: \"a617d026-0fd3-469b-87ec-47b0f4d85ad5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74aa8537-zxf2j"
Oct 30 22:08:44.739560 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:44.739530    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a617d026-0fd3-469b-87ec-47b0f4d85ad5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-74aa8537-zxf2j\" (UID: \"a617d026-0fd3-469b-87ec-47b0f4d85ad5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74aa8537-zxf2j"
Oct 30 22:08:44.740063 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:44.740041    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a617d026-0fd3-469b-87ec-47b0f4d85ad5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-74aa8537-zxf2j\" (UID: \"a617d026-0fd3-469b-87ec-47b0f4d85ad5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74aa8537-zxf2j"
Oct 30 22:08:44.741867 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:44.741845    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a617d026-0fd3-469b-87ec-47b0f4d85ad5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-74aa8537-zxf2j\" (UID: \"a617d026-0fd3-469b-87ec-47b0f4d85ad5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74aa8537-zxf2j"
Oct 30 22:08:44.797789 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:44.797729    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fm69d\" (UniqueName: \"kubernetes.io/projected/a617d026-0fd3-469b-87ec-47b0f4d85ad5-kube-api-access-fm69d\") pod \"maas-api-revoke-sub-74aa8537-zxf2j\" (UID: \"a617d026-0fd3-469b-87ec-47b0f4d85ad5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74aa8537-zxf2j"
Oct 30 22:08:44.861289 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:44.861261    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74aa8537-zxf2j"
Oct 30 22:08:45.181246 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:45.181218    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-gnkmq"]
Oct 30 22:08:45.185070 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:45.185050    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-gnkmq"
Oct 30 22:08:45.235006 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:45.234981    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-c9dbe6-openshift-default-dockercfg-bpxxl\""
Oct 30 22:08:45.241568 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:45.241541    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74aa8537-zxf2j"]
Oct 30 22:08:45.242480 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:08:45.242457    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda617d026_0fd3_469b_87ec_47b0f4d85ad5.slice/crio-1f4a49eb18c1c85d99ca099d23294fb553cf5b6cdb4e8f7b45b3c539c4fa6a39 WatchSource:0}: Error finding container 1f4a49eb18c1c85d99ca099d23294fb553cf5b6cdb4e8f7b45b3c539c4fa6a39: Status 404 returned error can't find the container with id 1f4a49eb18c1c85d99ca099d23294fb553cf5b6cdb4e8f7b45b3c539c4fa6a39
Oct 30 22:08:45.247878 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:45.247855    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58723455-ca0f-4dbe-ac85-325c6aa5d740-istio-envoy\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-gnkmq\" (UID: \"58723455-ca0f-4dbe-ac85-325c6aa5d740\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-gnkmq"
Oct 30 22:08:45.247984 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:45.247891    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58723455-ca0f-4dbe-ac85-325c6aa5d740-credential-socket\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-gnkmq\" (UID: \"58723455-ca0f-4dbe-ac85-325c6aa5d740\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-gnkmq"
Oct 30 22:08:45.247984 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:45.247917    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58723455-ca0f-4dbe-ac85-325c6aa5d740-istio-data\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-gnkmq\" (UID: \"58723455-ca0f-4dbe-ac85-325c6aa5d740\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-gnkmq"
Oct 30 22:08:45.248105 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:45.248081    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58723455-ca0f-4dbe-ac85-325c6aa5d740-workload-certs\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-gnkmq\" (UID: \"58723455-ca0f-4dbe-ac85-325c6aa5d740\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-gnkmq"
Oct 30 22:08:45.248166 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:45.248125    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58723455-ca0f-4dbe-ac85-325c6aa5d740-istio-podinfo\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-gnkmq\" (UID: \"58723455-ca0f-4dbe-ac85-325c6aa5d740\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-gnkmq"
Oct 30 22:08:45.248242 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:45.248222    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58723455-ca0f-4dbe-ac85-325c6aa5d740-istiod-ca-cert\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-gnkmq\" (UID: \"58723455-ca0f-4dbe-ac85-325c6aa5d740\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-gnkmq"
Oct 30 22:08:45.248313 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:45.248264    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58723455-ca0f-4dbe-ac85-325c6aa5d740-workload-socket\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-gnkmq\" (UID: \"58723455-ca0f-4dbe-ac85-325c6aa5d740\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-gnkmq"
Oct 30 22:08:45.248440 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:45.248419    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-994tj\" (UniqueName: \"kubernetes.io/projected/58723455-ca0f-4dbe-ac85-325c6aa5d740-kube-api-access-994tj\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-gnkmq\" (UID: \"58723455-ca0f-4dbe-ac85-325c6aa5d740\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-gnkmq"
Oct 30 22:08:45.248533 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:45.248468    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58723455-ca0f-4dbe-ac85-325c6aa5d740-istio-token\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-gnkmq\" (UID: \"58723455-ca0f-4dbe-ac85-325c6aa5d740\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-gnkmq"
Oct 30 22:08:45.303568 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:45.303524    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-gnkmq"]
Oct 30 22:08:45.349382 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:45.349351    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-994tj\" (UniqueName: \"kubernetes.io/projected/58723455-ca0f-4dbe-ac85-325c6aa5d740-kube-api-access-994tj\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-gnkmq\" (UID: \"58723455-ca0f-4dbe-ac85-325c6aa5d740\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-gnkmq"
Oct 30 22:08:45.349516 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:45.349391    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58723455-ca0f-4dbe-ac85-325c6aa5d740-istio-token\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-gnkmq\" (UID: \"58723455-ca0f-4dbe-ac85-325c6aa5d740\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-gnkmq"
Oct 30 22:08:45.349516 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:45.349472    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58723455-ca0f-4dbe-ac85-325c6aa5d740-istio-envoy\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-gnkmq\" (UID: \"58723455-ca0f-4dbe-ac85-325c6aa5d740\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-gnkmq"
Oct 30 22:08:45.349516 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:45.349498    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58723455-ca0f-4dbe-ac85-325c6aa5d740-credential-socket\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-gnkmq\" (UID: \"58723455-ca0f-4dbe-ac85-325c6aa5d740\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-gnkmq"
Oct 30 22:08:45.349516 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:45.349512    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58723455-ca0f-4dbe-ac85-325c6aa5d740-istio-data\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-gnkmq\" (UID: \"58723455-ca0f-4dbe-ac85-325c6aa5d740\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-gnkmq"
Oct 30 22:08:45.349688 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:45.349580    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58723455-ca0f-4dbe-ac85-325c6aa5d740-workload-certs\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-gnkmq\" (UID: \"58723455-ca0f-4dbe-ac85-325c6aa5d740\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-gnkmq"
Oct 30 22:08:45.349688 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:45.349608    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58723455-ca0f-4dbe-ac85-325c6aa5d740-istio-podinfo\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-gnkmq\" (UID: \"58723455-ca0f-4dbe-ac85-325c6aa5d740\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-gnkmq"
Oct 30 22:08:45.349688 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:45.349653    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58723455-ca0f-4dbe-ac85-325c6aa5d740-istiod-ca-cert\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-gnkmq\" (UID: \"58723455-ca0f-4dbe-ac85-325c6aa5d740\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-gnkmq"
Oct 30 22:08:45.349830 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:45.349699    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58723455-ca0f-4dbe-ac85-325c6aa5d740-workload-socket\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-gnkmq\" (UID: \"58723455-ca0f-4dbe-ac85-325c6aa5d740\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-gnkmq"
Oct 30 22:08:45.349990 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:45.349965    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58723455-ca0f-4dbe-ac85-325c6aa5d740-istio-data\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-gnkmq\" (UID: \"58723455-ca0f-4dbe-ac85-325c6aa5d740\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-gnkmq"
Oct 30 22:08:45.350066 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:45.350022    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58723455-ca0f-4dbe-ac85-325c6aa5d740-credential-socket\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-gnkmq\" (UID: \"58723455-ca0f-4dbe-ac85-325c6aa5d740\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-gnkmq"
Oct 30 22:08:45.350136 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:45.350116    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58723455-ca0f-4dbe-ac85-325c6aa5d740-workload-certs\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-gnkmq\" (UID: \"58723455-ca0f-4dbe-ac85-325c6aa5d740\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-gnkmq"
Oct 30 22:08:45.350212 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:45.350139    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58723455-ca0f-4dbe-ac85-325c6aa5d740-workload-socket\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-gnkmq\" (UID: \"58723455-ca0f-4dbe-ac85-325c6aa5d740\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-gnkmq"
Oct 30 22:08:45.350379 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:45.350357    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58723455-ca0f-4dbe-ac85-325c6aa5d740-istiod-ca-cert\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-gnkmq\" (UID: \"58723455-ca0f-4dbe-ac85-325c6aa5d740\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-gnkmq"
Oct 30 22:08:45.351656 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:45.351637    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58723455-ca0f-4dbe-ac85-325c6aa5d740-istio-envoy\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-gnkmq\" (UID: \"58723455-ca0f-4dbe-ac85-325c6aa5d740\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-gnkmq"
Oct 30 22:08:45.352004 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:45.351986    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58723455-ca0f-4dbe-ac85-325c6aa5d740-istio-podinfo\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-gnkmq\" (UID: \"58723455-ca0f-4dbe-ac85-325c6aa5d740\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-gnkmq"
Oct 30 22:08:45.408668 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:45.408642    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58723455-ca0f-4dbe-ac85-325c6aa5d740-istio-token\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-gnkmq\" (UID: \"58723455-ca0f-4dbe-ac85-325c6aa5d740\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-gnkmq"
Oct 30 22:08:45.408789 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:45.408739    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-994tj\" (UniqueName: \"kubernetes.io/projected/58723455-ca0f-4dbe-ac85-325c6aa5d740-kube-api-access-994tj\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-gnkmq\" (UID: \"58723455-ca0f-4dbe-ac85-325c6aa5d740\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-gnkmq"
Oct 30 22:08:45.499142 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:45.499114    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-gnkmq"
Oct 30 22:08:45.947136 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:45.947109    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-gnkmq"]
Oct 30 22:08:45.948928 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:08:45.948899    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58723455_ca0f_4dbe_ac85_325c6aa5d740.slice/crio-6324d6dfa23d3a2bf940e6fe26e9f77b7f4d28d9fe37e3e1c956418197ebb2d7 WatchSource:0}: Error finding container 6324d6dfa23d3a2bf940e6fe26e9f77b7f4d28d9fe37e3e1c956418197ebb2d7: Status 404 returned error can't find the container with id 6324d6dfa23d3a2bf940e6fe26e9f77b7f4d28d9fe37e3e1c956418197ebb2d7
Oct 30 22:08:45.951670 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:45.951637    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:08:45.951770 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:45.951718    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:08:45.951834 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:45.951766    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:08:46.139129 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:46.139104    5678 generic.go:356] "Generic (PLEG): container finished" podID="a617d026-0fd3-469b-87ec-47b0f4d85ad5" containerID="cc6d9e00b5fadbe1f0e6b9a1feef29fc005705fcf552bf8fe3aa35d9429e9e6c" exitCode=0
Oct 30 22:08:46.139243 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:46.139187    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74aa8537-zxf2j" event={"ID":"a617d026-0fd3-469b-87ec-47b0f4d85ad5","Type":"ContainerDied","Data":"cc6d9e00b5fadbe1f0e6b9a1feef29fc005705fcf552bf8fe3aa35d9429e9e6c"}
Oct 30 22:08:46.139315 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:46.139251    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74aa8537-zxf2j" event={"ID":"a617d026-0fd3-469b-87ec-47b0f4d85ad5","Type":"ContainerStarted","Data":"1f4a49eb18c1c85d99ca099d23294fb553cf5b6cdb4e8f7b45b3c539c4fa6a39"}
Oct 30 22:08:46.140993 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:46.140969    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-gnkmq" event={"ID":"58723455-ca0f-4dbe-ac85-325c6aa5d740","Type":"ContainerStarted","Data":"09be238885d49a245bc397a3745dcf18a77fdf3e1b7d70ed614deb840b32d988"}
Oct 30 22:08:46.141080 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:46.141015    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-gnkmq" event={"ID":"58723455-ca0f-4dbe-ac85-325c6aa5d740","Type":"ContainerStarted","Data":"6324d6dfa23d3a2bf940e6fe26e9f77b7f4d28d9fe37e3e1c956418197ebb2d7"}
Oct 30 22:08:46.366724 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:46.366678    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-gnkmq" podStartSLOduration=1.366663888 podStartE2EDuration="1.366663888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:08:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:08:46.365784019 +0000 UTC m=+1103.970091586" watchObservedRunningTime="2026-10-30 22:08:46.366663888 +0000 UTC m=+1103.970971441"
Oct 30 22:08:46.499359 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:46.499327    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-gnkmq"
Oct 30 22:08:47.312241 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:47.312214    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74aa8537-zxf2j"
Oct 30 22:08:47.375000 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:47.374964    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a617d026-0fd3-469b-87ec-47b0f4d85ad5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a617d026-0fd3-469b-87ec-47b0f4d85ad5-maas-api-openshift-ca\") pod \"a617d026-0fd3-469b-87ec-47b0f4d85ad5\" (UID: \"a617d026-0fd3-469b-87ec-47b0f4d85ad5\") "
Oct 30 22:08:47.375231 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:47.375143    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a617d026-0fd3-469b-87ec-47b0f4d85ad5-kube-api-access-fm69d\" (UniqueName: \"kubernetes.io/projected/a617d026-0fd3-469b-87ec-47b0f4d85ad5-kube-api-access-fm69d\") pod \"a617d026-0fd3-469b-87ec-47b0f4d85ad5\" (UID: \"a617d026-0fd3-469b-87ec-47b0f4d85ad5\") "
Oct 30 22:08:47.375231 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:47.375192    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a617d026-0fd3-469b-87ec-47b0f4d85ad5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a617d026-0fd3-469b-87ec-47b0f4d85ad5-maas-api-xks-ca\") pod \"a617d026-0fd3-469b-87ec-47b0f4d85ad5\" (UID: \"a617d026-0fd3-469b-87ec-47b0f4d85ad5\") "
Oct 30 22:08:47.375446 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:47.375353    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a617d026-0fd3-469b-87ec-47b0f4d85ad5-maas-api-openshift-ca" pod "a617d026-0fd3-469b-87ec-47b0f4d85ad5" (UID: "a617d026-0fd3-469b-87ec-47b0f4d85ad5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:08:47.375672 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:47.375650    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a617d026-0fd3-469b-87ec-47b0f4d85ad5-maas-api-openshift-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:47.377673 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:47.377643    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a617d026-0fd3-469b-87ec-47b0f4d85ad5-maas-api-xks-ca" pod "a617d026-0fd3-469b-87ec-47b0f4d85ad5" (UID: "a617d026-0fd3-469b-87ec-47b0f4d85ad5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:08:47.378132 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:47.378114    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a617d026-0fd3-469b-87ec-47b0f4d85ad5-kube-api-access-fm69d" pod "a617d026-0fd3-469b-87ec-47b0f4d85ad5" (UID: "a617d026-0fd3-469b-87ec-47b0f4d85ad5"). InnerVolumeSpecName "kube-api-access-fm69d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:08:47.476702 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:47.476607    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fm69d\" (UniqueName: \"kubernetes.io/projected/a617d026-0fd3-469b-87ec-47b0f4d85ad5-kube-api-access-fm69d\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:47.476702 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:47.476645    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a617d026-0fd3-469b-87ec-47b0f4d85ad5-maas-api-xks-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:47.501256 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:47.501224    5678 patch_prober.go:29] interesting pod/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-gnkmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Oct 30 22:08:47.501422 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:47.501275    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-gnkmq" podUID="58723455-ca0f-4dbe-ac85-325c6aa5d740" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Oct 30 22:08:48.159733 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:48.159703    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74aa8537-zxf2j"
Oct 30 22:08:48.159733 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:48.159719    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74aa8537-zxf2j" event={"ID":"a617d026-0fd3-469b-87ec-47b0f4d85ad5","Type":"ContainerDied","Data":"1f4a49eb18c1c85d99ca099d23294fb553cf5b6cdb4e8f7b45b3c539c4fa6a39"}
Oct 30 22:08:48.159979 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:48.159769    5678 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1f4a49eb18c1c85d99ca099d23294fb553cf5b6cdb4e8f7b45b3c539c4fa6a39"
Oct 30 22:08:48.499429 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:48.499345    5678 patch_prober.go:29] interesting pod/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-gnkmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Oct 30 22:08:48.499792 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:48.499424    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-gnkmq" podUID="58723455-ca0f-4dbe-ac85-325c6aa5d740" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Oct 30 22:08:49.499610 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:49.499581    5678 patch_prober.go:29] interesting pod/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-gnkmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Oct 30 22:08:49.499987 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:49.499626    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-gnkmq" podUID="58723455-ca0f-4dbe-ac85-325c6aa5d740" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Oct 30 22:08:50.499672 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:50.499637    5678 patch_prober.go:29] interesting pod/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-gnkmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Oct 30 22:08:50.500070 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:50.499692    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-gnkmq" podUID="58723455-ca0f-4dbe-ac85-325c6aa5d740" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Oct 30 22:08:50.692315 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:50.692279    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e429ea7c-trt9p"]
Oct 30 22:08:50.692837 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:50.692824    5678 cpu_manager.go:414] "containerMap: removing container" podUID="a617d026-0fd3-469b-87ec-47b0f4d85ad5" containerName="revoke-keys"
Oct 30 22:08:50.692881 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:50.692838    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a617d026-0fd3-469b-87ec-47b0f4d85ad5" containerName="revoke-keys"
Oct 30 22:08:50.692939 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:50.692930    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="a617d026-0fd3-469b-87ec-47b0f4d85ad5" containerName="revoke-keys"
Oct 30 22:08:50.696183 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:50.696158    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e429ea7c-trt9p"
Oct 30 22:08:50.756727 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:50.756567    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e429ea7c-trt9p"]
Oct 30 22:08:50.798777 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:50.798750    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5rvcj\""
Oct 30 22:08:50.798948 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:50.798755    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:08:50.812832 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:50.812807    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65e57873-3428-4f06-90f7-d8e0a49c0f9e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e429ea7c-trt9p\" (UID: \"65e57873-3428-4f06-90f7-d8e0a49c0f9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e429ea7c-trt9p"
Oct 30 22:08:50.812955 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:50.812850    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcqfp\" (UniqueName: \"kubernetes.io/projected/65e57873-3428-4f06-90f7-d8e0a49c0f9e-kube-api-access-dcqfp\") pod \"maas-api-revoke-sub-e429ea7c-trt9p\" (UID: \"65e57873-3428-4f06-90f7-d8e0a49c0f9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e429ea7c-trt9p"
Oct 30 22:08:50.812955 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:50.812936    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65e57873-3428-4f06-90f7-d8e0a49c0f9e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e429ea7c-trt9p\" (UID: \"65e57873-3428-4f06-90f7-d8e0a49c0f9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e429ea7c-trt9p"
Oct 30 22:08:50.913639 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:50.913601    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65e57873-3428-4f06-90f7-d8e0a49c0f9e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e429ea7c-trt9p\" (UID: \"65e57873-3428-4f06-90f7-d8e0a49c0f9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e429ea7c-trt9p"
Oct 30 22:08:50.913807 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:50.913651    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcqfp\" (UniqueName: \"kubernetes.io/projected/65e57873-3428-4f06-90f7-d8e0a49c0f9e-kube-api-access-dcqfp\") pod \"maas-api-revoke-sub-e429ea7c-trt9p\" (UID: \"65e57873-3428-4f06-90f7-d8e0a49c0f9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e429ea7c-trt9p"
Oct 30 22:08:50.913807 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:50.913691    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65e57873-3428-4f06-90f7-d8e0a49c0f9e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e429ea7c-trt9p\" (UID: \"65e57873-3428-4f06-90f7-d8e0a49c0f9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e429ea7c-trt9p"
Oct 30 22:08:50.914255 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:50.914231    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65e57873-3428-4f06-90f7-d8e0a49c0f9e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e429ea7c-trt9p\" (UID: \"65e57873-3428-4f06-90f7-d8e0a49c0f9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e429ea7c-trt9p"
Oct 30 22:08:50.915871 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:50.915821    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65e57873-3428-4f06-90f7-d8e0a49c0f9e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e429ea7c-trt9p\" (UID: \"65e57873-3428-4f06-90f7-d8e0a49c0f9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e429ea7c-trt9p"
Oct 30 22:08:50.972795 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:50.972759    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcqfp\" (UniqueName: \"kubernetes.io/projected/65e57873-3428-4f06-90f7-d8e0a49c0f9e-kube-api-access-dcqfp\") pod \"maas-api-revoke-sub-e429ea7c-trt9p\" (UID: \"65e57873-3428-4f06-90f7-d8e0a49c0f9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e429ea7c-trt9p"
Oct 30 22:08:51.010013 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:51.009941    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e429ea7c-trt9p"
Oct 30 22:08:51.293846 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:51.293764    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebb90d20-csn9b"]
Oct 30 22:08:51.297712 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:51.297690    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebb90d20-csn9b"
Oct 30 22:08:51.368055 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:51.368006    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebb90d20-csn9b"]
Oct 30 22:08:51.421083 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:51.421049    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d0dc96c-49eb-4b32-9154-d512fcef9f05-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ebb90d20-csn9b\" (UID: \"4d0dc96c-49eb-4b32-9154-d512fcef9f05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebb90d20-csn9b"
Oct 30 22:08:51.421254 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:51.421127    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d0dc96c-49eb-4b32-9154-d512fcef9f05-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ebb90d20-csn9b\" (UID: \"4d0dc96c-49eb-4b32-9154-d512fcef9f05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebb90d20-csn9b"
Oct 30 22:08:51.421339 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:51.421315    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcx8n\" (UniqueName: \"kubernetes.io/projected/4d0dc96c-49eb-4b32-9154-d512fcef9f05-kube-api-access-qcx8n\") pod \"maas-api-revoke-sub-ebb90d20-csn9b\" (UID: \"4d0dc96c-49eb-4b32-9154-d512fcef9f05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebb90d20-csn9b"
Oct 30 22:08:51.473535 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:51.473499    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e429ea7c-trt9p"]
Oct 30 22:08:51.474474 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:08:51.474445    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65e57873_3428_4f06_90f7_d8e0a49c0f9e.slice/crio-16898e029ad6f4b00381342db30efdbbe0965d4428b0aeed7bb104f1a011d8cd WatchSource:0}: Error finding container 16898e029ad6f4b00381342db30efdbbe0965d4428b0aeed7bb104f1a011d8cd: Status 404 returned error can't find the container with id 16898e029ad6f4b00381342db30efdbbe0965d4428b0aeed7bb104f1a011d8cd
Oct 30 22:08:51.499927 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:51.499902    5678 patch_prober.go:29] interesting pod/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-gnkmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Oct 30 22:08:51.500245 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:51.499956    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-gnkmq" podUID="58723455-ca0f-4dbe-ac85-325c6aa5d740" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Oct 30 22:08:51.542721 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:51.542682    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d0dc96c-49eb-4b32-9154-d512fcef9f05-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ebb90d20-csn9b\" (UID: \"4d0dc96c-49eb-4b32-9154-d512fcef9f05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebb90d20-csn9b"
Oct 30 22:08:51.542928 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:51.542908    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d0dc96c-49eb-4b32-9154-d512fcef9f05-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ebb90d20-csn9b\" (UID: \"4d0dc96c-49eb-4b32-9154-d512fcef9f05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebb90d20-csn9b"
Oct 30 22:08:51.543177 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:51.543159    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcx8n\" (UniqueName: \"kubernetes.io/projected/4d0dc96c-49eb-4b32-9154-d512fcef9f05-kube-api-access-qcx8n\") pod \"maas-api-revoke-sub-ebb90d20-csn9b\" (UID: \"4d0dc96c-49eb-4b32-9154-d512fcef9f05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebb90d20-csn9b"
Oct 30 22:08:51.549367 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:51.544616    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d0dc96c-49eb-4b32-9154-d512fcef9f05-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ebb90d20-csn9b\" (UID: \"4d0dc96c-49eb-4b32-9154-d512fcef9f05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebb90d20-csn9b"
Oct 30 22:08:51.646968 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:51.646937    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d0dc96c-49eb-4b32-9154-d512fcef9f05-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ebb90d20-csn9b\" (UID: \"4d0dc96c-49eb-4b32-9154-d512fcef9f05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebb90d20-csn9b"
Oct 30 22:08:51.647583 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:51.647562    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcx8n\" (UniqueName: \"kubernetes.io/projected/4d0dc96c-49eb-4b32-9154-d512fcef9f05-kube-api-access-qcx8n\") pod \"maas-api-revoke-sub-ebb90d20-csn9b\" (UID: \"4d0dc96c-49eb-4b32-9154-d512fcef9f05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebb90d20-csn9b"
Oct 30 22:08:51.911621 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:51.911591    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebb90d20-csn9b"
Oct 30 22:08:52.186953 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:52.186917    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e429ea7c-trt9p" event={"ID":"65e57873-3428-4f06-90f7-d8e0a49c0f9e","Type":"ContainerStarted","Data":"a4fe8669512485d9b64a520c0fd490748e35be08af5c992e466808be94e56777"}
Oct 30 22:08:52.187211 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:52.186978    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e429ea7c-trt9p" event={"ID":"65e57873-3428-4f06-90f7-d8e0a49c0f9e","Type":"ContainerStarted","Data":"16898e029ad6f4b00381342db30efdbbe0965d4428b0aeed7bb104f1a011d8cd"}
Oct 30 22:08:52.301140 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:52.301063    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e429ea7c-trt9p" podStartSLOduration=2.301044178 podStartE2EDuration="2.301044178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:08:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:08:52.297349876 +0000 UTC m=+1109.901657430" watchObservedRunningTime="2026-10-30 22:08:52.301044178 +0000 UTC m=+1109.905351733"
Oct 30 22:08:52.306857 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:08:52.306830    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d0dc96c_49eb_4b32_9154_d512fcef9f05.slice/crio-deccb36b2f0980a10c251f9f9725685c75816d429017f21391e8f6ffb3f15c6c WatchSource:0}: Error finding container deccb36b2f0980a10c251f9f9725685c75816d429017f21391e8f6ffb3f15c6c: Status 404 returned error can't find the container with id deccb36b2f0980a10c251f9f9725685c75816d429017f21391e8f6ffb3f15c6c
Oct 30 22:08:52.307382 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:52.307357    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebb90d20-csn9b"]
Oct 30 22:08:52.684816 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:52.684792    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-gnkmq"
Oct 30 22:08:52.685284 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:52.685260    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-gnkmq"
Oct 30 22:08:52.686144 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:52.686124    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-gnkmq"
Oct 30 22:08:53.195894 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:53.195866    5678 generic.go:356] "Generic (PLEG): container finished" podID="65e57873-3428-4f06-90f7-d8e0a49c0f9e" containerID="a4fe8669512485d9b64a520c0fd490748e35be08af5c992e466808be94e56777" exitCode=0
Oct 30 22:08:53.196010 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:53.195941    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e429ea7c-trt9p" event={"ID":"65e57873-3428-4f06-90f7-d8e0a49c0f9e","Type":"ContainerDied","Data":"a4fe8669512485d9b64a520c0fd490748e35be08af5c992e466808be94e56777"}
Oct 30 22:08:53.198491 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:53.198464    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebb90d20-csn9b" event={"ID":"4d0dc96c-49eb-4b32-9154-d512fcef9f05","Type":"ContainerStarted","Data":"8225b867fc1f02c4c04f99fdbf7bd003b338a1e0a6cea99ec4d83fe5fca3f10d"}
Oct 30 22:08:53.198596 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:53.198511    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebb90d20-csn9b" event={"ID":"4d0dc96c-49eb-4b32-9154-d512fcef9f05","Type":"ContainerStarted","Data":"deccb36b2f0980a10c251f9f9725685c75816d429017f21391e8f6ffb3f15c6c"}
Oct 30 22:08:54.205932 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:54.205895    5678 generic.go:356] "Generic (PLEG): container finished" podID="4d0dc96c-49eb-4b32-9154-d512fcef9f05" containerID="8225b867fc1f02c4c04f99fdbf7bd003b338a1e0a6cea99ec4d83fe5fca3f10d" exitCode=0
Oct 30 22:08:54.206357 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:54.205977    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebb90d20-csn9b" event={"ID":"4d0dc96c-49eb-4b32-9154-d512fcef9f05","Type":"ContainerDied","Data":"8225b867fc1f02c4c04f99fdbf7bd003b338a1e0a6cea99ec4d83fe5fca3f10d"}
Oct 30 22:08:54.371425 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:54.371390    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebb90d20-csn9b"
Oct 30 22:08:54.374829 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:54.374813    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e429ea7c-trt9p"
Oct 30 22:08:54.481394 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:54.481321    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d0dc96c-49eb-4b32-9154-d512fcef9f05-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d0dc96c-49eb-4b32-9154-d512fcef9f05-maas-api-xks-ca\") pod \"4d0dc96c-49eb-4b32-9154-d512fcef9f05\" (UID: \"4d0dc96c-49eb-4b32-9154-d512fcef9f05\") "
Oct 30 22:08:54.481544 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:54.481430    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d0dc96c-49eb-4b32-9154-d512fcef9f05-kube-api-access-qcx8n\" (UniqueName: \"kubernetes.io/projected/4d0dc96c-49eb-4b32-9154-d512fcef9f05-kube-api-access-qcx8n\") pod \"4d0dc96c-49eb-4b32-9154-d512fcef9f05\" (UID: \"4d0dc96c-49eb-4b32-9154-d512fcef9f05\") "
Oct 30 22:08:54.481544 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:54.481474    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65e57873-3428-4f06-90f7-d8e0a49c0f9e-kube-api-access-dcqfp\" (UniqueName: \"kubernetes.io/projected/65e57873-3428-4f06-90f7-d8e0a49c0f9e-kube-api-access-dcqfp\") pod \"65e57873-3428-4f06-90f7-d8e0a49c0f9e\" (UID: \"65e57873-3428-4f06-90f7-d8e0a49c0f9e\") "
Oct 30 22:08:54.481544 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:54.481525    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65e57873-3428-4f06-90f7-d8e0a49c0f9e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65e57873-3428-4f06-90f7-d8e0a49c0f9e-maas-api-openshift-ca\") pod \"65e57873-3428-4f06-90f7-d8e0a49c0f9e\" (UID: \"65e57873-3428-4f06-90f7-d8e0a49c0f9e\") "
Oct 30 22:08:54.481711 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:54.481606    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d0dc96c-49eb-4b32-9154-d512fcef9f05-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d0dc96c-49eb-4b32-9154-d512fcef9f05-maas-api-openshift-ca\") pod \"4d0dc96c-49eb-4b32-9154-d512fcef9f05\" (UID: \"4d0dc96c-49eb-4b32-9154-d512fcef9f05\") "
Oct 30 22:08:54.481711 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:54.481686    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65e57873-3428-4f06-90f7-d8e0a49c0f9e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65e57873-3428-4f06-90f7-d8e0a49c0f9e-maas-api-xks-ca\") pod \"65e57873-3428-4f06-90f7-d8e0a49c0f9e\" (UID: \"65e57873-3428-4f06-90f7-d8e0a49c0f9e\") "
Oct 30 22:08:54.482318 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:54.482092    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65e57873-3428-4f06-90f7-d8e0a49c0f9e-maas-api-openshift-ca" pod "65e57873-3428-4f06-90f7-d8e0a49c0f9e" (UID: "65e57873-3428-4f06-90f7-d8e0a49c0f9e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:08:54.482318 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:54.482184    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65e57873-3428-4f06-90f7-d8e0a49c0f9e-maas-api-openshift-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:54.482593 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:54.482565    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d0dc96c-49eb-4b32-9154-d512fcef9f05-maas-api-openshift-ca" pod "4d0dc96c-49eb-4b32-9154-d512fcef9f05" (UID: "4d0dc96c-49eb-4b32-9154-d512fcef9f05"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:08:54.484072 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:54.484035    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d0dc96c-49eb-4b32-9154-d512fcef9f05-maas-api-xks-ca" pod "4d0dc96c-49eb-4b32-9154-d512fcef9f05" (UID: "4d0dc96c-49eb-4b32-9154-d512fcef9f05"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:08:54.484169 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:54.484143    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65e57873-3428-4f06-90f7-d8e0a49c0f9e-kube-api-access-dcqfp" pod "65e57873-3428-4f06-90f7-d8e0a49c0f9e" (UID: "65e57873-3428-4f06-90f7-d8e0a49c0f9e"). InnerVolumeSpecName "kube-api-access-dcqfp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:08:54.484423 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:54.484370    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65e57873-3428-4f06-90f7-d8e0a49c0f9e-maas-api-xks-ca" pod "65e57873-3428-4f06-90f7-d8e0a49c0f9e" (UID: "65e57873-3428-4f06-90f7-d8e0a49c0f9e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:08:54.484512 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:54.484425    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d0dc96c-49eb-4b32-9154-d512fcef9f05-kube-api-access-qcx8n" pod "4d0dc96c-49eb-4b32-9154-d512fcef9f05" (UID: "4d0dc96c-49eb-4b32-9154-d512fcef9f05"). InnerVolumeSpecName "kube-api-access-qcx8n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:08:54.592286 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:54.592054    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d0dc96c-49eb-4b32-9154-d512fcef9f05-maas-api-openshift-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:54.592286 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:54.592088    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65e57873-3428-4f06-90f7-d8e0a49c0f9e-maas-api-xks-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:54.592286 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:54.592103    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d0dc96c-49eb-4b32-9154-d512fcef9f05-maas-api-xks-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:54.592286 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:54.592118    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcx8n\" (UniqueName: \"kubernetes.io/projected/4d0dc96c-49eb-4b32-9154-d512fcef9f05-kube-api-access-qcx8n\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:54.592286 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:54.592132    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcqfp\" (UniqueName: \"kubernetes.io/projected/65e57873-3428-4f06-90f7-d8e0a49c0f9e-kube-api-access-dcqfp\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:54.619105 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:54.619066    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-9vxbk"]
Oct 30 22:08:54.620094 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:54.620068    5678 cpu_manager.go:414] "containerMap: removing container" podUID="65e57873-3428-4f06-90f7-d8e0a49c0f9e" containerName="revoke-keys"
Oct 30 22:08:54.620229 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:54.620114    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65e57873-3428-4f06-90f7-d8e0a49c0f9e" containerName="revoke-keys"
Oct 30 22:08:54.620229 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:54.620142    5678 cpu_manager.go:414] "containerMap: removing container" podUID="4d0dc96c-49eb-4b32-9154-d512fcef9f05" containerName="revoke-keys"
Oct 30 22:08:54.620229 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:54.620152    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d0dc96c-49eb-4b32-9154-d512fcef9f05" containerName="revoke-keys"
Oct 30 22:08:54.620388 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:54.620357    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="65e57873-3428-4f06-90f7-d8e0a49c0f9e" containerName="revoke-keys"
Oct 30 22:08:54.620388 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:54.620372    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d0dc96c-49eb-4b32-9154-d512fcef9f05" containerName="revoke-keys"
Oct 30 22:08:54.638053 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:54.638022    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-9vxbk"
Oct 30 22:08:54.739784 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:54.739653    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-9vxbk"]
Oct 30 22:08:54.794046 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:54.794004    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bde7f877-4938-42d4-be6c-cba04795ce83-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-9vxbk\" (UID: \"bde7f877-4938-42d4-be6c-cba04795ce83\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-9vxbk"
Oct 30 22:08:54.794216 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:54.794058    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bde7f877-4938-42d4-be6c-cba04795ce83-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-9vxbk\" (UID: \"bde7f877-4938-42d4-be6c-cba04795ce83\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-9vxbk"
Oct 30 22:08:54.794216 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:54.794162    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bde7f877-4938-42d4-be6c-cba04795ce83-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-9vxbk\" (UID: \"bde7f877-4938-42d4-be6c-cba04795ce83\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-9vxbk"
Oct 30 22:08:54.794216 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:54.794195    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bde7f877-4938-42d4-be6c-cba04795ce83-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-9vxbk\" (UID: \"bde7f877-4938-42d4-be6c-cba04795ce83\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-9vxbk"
Oct 30 22:08:54.794339 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:54.794229    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bde7f877-4938-42d4-be6c-cba04795ce83-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-9vxbk\" (UID: \"bde7f877-4938-42d4-be6c-cba04795ce83\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-9vxbk"
Oct 30 22:08:54.794339 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:54.794293    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mx68z\" (UniqueName: \"kubernetes.io/projected/bde7f877-4938-42d4-be6c-cba04795ce83-kube-api-access-mx68z\") pod \"maas-default-gateway-openshift-default-85544d4cd7-9vxbk\" (UID: \"bde7f877-4938-42d4-be6c-cba04795ce83\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-9vxbk"
Oct 30 22:08:54.794339 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:54.794317    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bde7f877-4938-42d4-be6c-cba04795ce83-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-9vxbk\" (UID: \"bde7f877-4938-42d4-be6c-cba04795ce83\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-9vxbk"
Oct 30 22:08:54.794475 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:54.794353    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bde7f877-4938-42d4-be6c-cba04795ce83-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-9vxbk\" (UID: \"bde7f877-4938-42d4-be6c-cba04795ce83\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-9vxbk"
Oct 30 22:08:54.794573 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:54.794553    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bde7f877-4938-42d4-be6c-cba04795ce83-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-9vxbk\" (UID: \"bde7f877-4938-42d4-be6c-cba04795ce83\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-9vxbk"
Oct 30 22:08:54.896422 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:54.896096    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bde7f877-4938-42d4-be6c-cba04795ce83-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-9vxbk\" (UID: \"bde7f877-4938-42d4-be6c-cba04795ce83\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-9vxbk"
Oct 30 22:08:54.897219 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:54.896588    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bde7f877-4938-42d4-be6c-cba04795ce83-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-9vxbk\" (UID: \"bde7f877-4938-42d4-be6c-cba04795ce83\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-9vxbk"
Oct 30 22:08:54.897219 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:54.896675    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bde7f877-4938-42d4-be6c-cba04795ce83-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-9vxbk\" (UID: \"bde7f877-4938-42d4-be6c-cba04795ce83\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-9vxbk"
Oct 30 22:08:54.897219 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:54.896702    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bde7f877-4938-42d4-be6c-cba04795ce83-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-9vxbk\" (UID: \"bde7f877-4938-42d4-be6c-cba04795ce83\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-9vxbk"
Oct 30 22:08:54.897219 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:54.896750    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bde7f877-4938-42d4-be6c-cba04795ce83-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-9vxbk\" (UID: \"bde7f877-4938-42d4-be6c-cba04795ce83\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-9vxbk"
Oct 30 22:08:54.897219 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:54.896781    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bde7f877-4938-42d4-be6c-cba04795ce83-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-9vxbk\" (UID: \"bde7f877-4938-42d4-be6c-cba04795ce83\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-9vxbk"
Oct 30 22:08:54.897219 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:54.896820    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bde7f877-4938-42d4-be6c-cba04795ce83-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-9vxbk\" (UID: \"bde7f877-4938-42d4-be6c-cba04795ce83\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-9vxbk"
Oct 30 22:08:54.897219 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:54.896870    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mx68z\" (UniqueName: \"kubernetes.io/projected/bde7f877-4938-42d4-be6c-cba04795ce83-kube-api-access-mx68z\") pod \"maas-default-gateway-openshift-default-85544d4cd7-9vxbk\" (UID: \"bde7f877-4938-42d4-be6c-cba04795ce83\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-9vxbk"
Oct 30 22:08:54.897219 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:54.896912    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bde7f877-4938-42d4-be6c-cba04795ce83-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-9vxbk\" (UID: \"bde7f877-4938-42d4-be6c-cba04795ce83\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-9vxbk"
Oct 30 22:08:54.897219 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:54.896979    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bde7f877-4938-42d4-be6c-cba04795ce83-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-9vxbk\" (UID: \"bde7f877-4938-42d4-be6c-cba04795ce83\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-9vxbk"
Oct 30 22:08:54.899140 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:54.899082    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bde7f877-4938-42d4-be6c-cba04795ce83-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-9vxbk\" (UID: \"bde7f877-4938-42d4-be6c-cba04795ce83\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-9vxbk"
Oct 30 22:08:54.903136 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:54.899829    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bde7f877-4938-42d4-be6c-cba04795ce83-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-9vxbk\" (UID: \"bde7f877-4938-42d4-be6c-cba04795ce83\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-9vxbk"
Oct 30 22:08:54.903258 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:54.900503    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bde7f877-4938-42d4-be6c-cba04795ce83-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-9vxbk\" (UID: \"bde7f877-4938-42d4-be6c-cba04795ce83\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-9vxbk"
Oct 30 22:08:54.903366 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:54.900710    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bde7f877-4938-42d4-be6c-cba04795ce83-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-9vxbk\" (UID: \"bde7f877-4938-42d4-be6c-cba04795ce83\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-9vxbk"
Oct 30 22:08:54.904488 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:54.904453    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bde7f877-4938-42d4-be6c-cba04795ce83-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-9vxbk\" (UID: \"bde7f877-4938-42d4-be6c-cba04795ce83\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-9vxbk"
Oct 30 22:08:54.906959 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:54.906936    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bde7f877-4938-42d4-be6c-cba04795ce83-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-9vxbk\" (UID: \"bde7f877-4938-42d4-be6c-cba04795ce83\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-9vxbk"
Oct 30 22:08:54.964470 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:54.964432    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mx68z\" (UniqueName: \"kubernetes.io/projected/bde7f877-4938-42d4-be6c-cba04795ce83-kube-api-access-mx68z\") pod \"maas-default-gateway-openshift-default-85544d4cd7-9vxbk\" (UID: \"bde7f877-4938-42d4-be6c-cba04795ce83\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-9vxbk"
Oct 30 22:08:54.964626 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:54.964434    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bde7f877-4938-42d4-be6c-cba04795ce83-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-9vxbk\" (UID: \"bde7f877-4938-42d4-be6c-cba04795ce83\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-9vxbk"
Oct 30 22:08:55.215779 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:55.215747    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebb90d20-csn9b"
Oct 30 22:08:55.215779 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:55.215755    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebb90d20-csn9b" event={"ID":"4d0dc96c-49eb-4b32-9154-d512fcef9f05","Type":"ContainerDied","Data":"deccb36b2f0980a10c251f9f9725685c75816d429017f21391e8f6ffb3f15c6c"}
Oct 30 22:08:55.216455 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:55.215808    5678 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="deccb36b2f0980a10c251f9f9725685c75816d429017f21391e8f6ffb3f15c6c"
Oct 30 22:08:55.218704 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:55.218679    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e429ea7c-trt9p" event={"ID":"65e57873-3428-4f06-90f7-d8e0a49c0f9e","Type":"ContainerDied","Data":"16898e029ad6f4b00381342db30efdbbe0965d4428b0aeed7bb104f1a011d8cd"}
Oct 30 22:08:55.218962 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:55.218947    5678 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="16898e029ad6f4b00381342db30efdbbe0965d4428b0aeed7bb104f1a011d8cd"
Oct 30 22:08:55.219070 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:55.218868    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e429ea7c-trt9p"
Oct 30 22:08:55.267423 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:55.267212    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-9vxbk"
Oct 30 22:08:56.502959 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:56.479560    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7bb1c2/premium-w3-7bb1c2-kserve-7dbf875d88-dqxn5" podUID="a58c911f-e2eb-4820-a6b6-24afe78b5fd4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Oct 30 22:08:56.502959 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:56.483182    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9ndd9" podUID="81bd6415-f42f-446b-a64e-956f794b3956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:08:56.580425 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:08:56.570992    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbde7f877_4938_42d4_be6c_cba04795ce83.slice/crio-99f73a7de5fb3c5c88d28ff7d650bceeb57930e2af6a19d8cd16f573f6f46c7c WatchSource:0}: Error finding container 99f73a7de5fb3c5c88d28ff7d650bceeb57930e2af6a19d8cd16f573f6f46c7c: Status 404 returned error can't find the container with id 99f73a7de5fb3c5c88d28ff7d650bceeb57930e2af6a19d8cd16f573f6f46c7c
Oct 30 22:08:56.594263 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:56.586225    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-9vxbk"]
Oct 30 22:08:56.597860 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:56.597819    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:08:56.597983 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:56.597956    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:08:56.598052 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:56.598011    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:08:57.393428 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:57.393379    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-9vxbk" event={"ID":"bde7f877-4938-42d4-be6c-cba04795ce83","Type":"ContainerStarted","Data":"026096dcf8dc7c8e0a89f9b9e951f419a9891095ba4a068e9cffe23d108aa87e"}
Oct 30 22:08:57.393612 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:57.393451    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-9vxbk" event={"ID":"bde7f877-4938-42d4-be6c-cba04795ce83","Type":"ContainerStarted","Data":"99f73a7de5fb3c5c88d28ff7d650bceeb57930e2af6a19d8cd16f573f6f46c7c"}
Oct 30 22:08:57.558743 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:57.558693    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-9vxbk" podStartSLOduration=3.55867682 podStartE2EDuration="3.55867682s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:08:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:08:57.555993592 +0000 UTC m=+1115.160301191" watchObservedRunningTime="2026-10-30 22:08:57.55867682 +0000 UTC m=+1115.162984374"
Oct 30 22:08:58.268380 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:58.268346    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-9vxbk"
Oct 30 22:08:58.270157 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:58.270131    5678 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-9vxbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Oct 30 22:08:58.270284 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:58.270200    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-9vxbk" podUID="bde7f877-4938-42d4-be6c-cba04795ce83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Oct 30 22:08:59.268797 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:59.268761    5678 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-9vxbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Oct 30 22:08:59.269203 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:08:59.268824    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-9vxbk" podUID="bde7f877-4938-42d4-be6c-cba04795ce83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Oct 30 22:09:00.268132 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:00.268101    5678 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-9vxbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Oct 30 22:09:00.268338 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:00.268170    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-9vxbk" podUID="bde7f877-4938-42d4-be6c-cba04795ce83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Oct 30 22:09:01.267844 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:01.267807    5678 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-9vxbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Oct 30 22:09:01.268339 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:01.267868    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-9vxbk" podUID="bde7f877-4938-42d4-be6c-cba04795ce83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Oct 30 22:09:02.109078 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:02.109049    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e648ccdf-5x4w5"]
Oct 30 22:09:02.112283 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:02.112261    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e648ccdf-5x4w5"
Oct 30 22:09:02.152600 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:02.152568    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58553747-132d-4758-ad10-8df667f1c29d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e648ccdf-5x4w5\" (UID: \"58553747-132d-4758-ad10-8df667f1c29d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e648ccdf-5x4w5"
Oct 30 22:09:02.152755 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:02.152638    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2n858\" (UniqueName: \"kubernetes.io/projected/58553747-132d-4758-ad10-8df667f1c29d-kube-api-access-2n858\") pod \"maas-api-revoke-sub-e648ccdf-5x4w5\" (UID: \"58553747-132d-4758-ad10-8df667f1c29d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e648ccdf-5x4w5"
Oct 30 22:09:02.152755 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:02.152716    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58553747-132d-4758-ad10-8df667f1c29d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e648ccdf-5x4w5\" (UID: \"58553747-132d-4758-ad10-8df667f1c29d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e648ccdf-5x4w5"
Oct 30 22:09:02.168728 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:02.168688    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e648ccdf-5x4w5"]
Oct 30 22:09:02.211690 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:02.211658    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:09:02.211847 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:02.211715    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5rvcj\""
Oct 30 22:09:02.254572 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:02.254535    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58553747-132d-4758-ad10-8df667f1c29d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e648ccdf-5x4w5\" (UID: \"58553747-132d-4758-ad10-8df667f1c29d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e648ccdf-5x4w5"
Oct 30 22:09:02.254750 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:02.254623    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2n858\" (UniqueName: \"kubernetes.io/projected/58553747-132d-4758-ad10-8df667f1c29d-kube-api-access-2n858\") pod \"maas-api-revoke-sub-e648ccdf-5x4w5\" (UID: \"58553747-132d-4758-ad10-8df667f1c29d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e648ccdf-5x4w5"
Oct 30 22:09:02.254750 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:02.254658    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58553747-132d-4758-ad10-8df667f1c29d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e648ccdf-5x4w5\" (UID: \"58553747-132d-4758-ad10-8df667f1c29d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e648ccdf-5x4w5"
Oct 30 22:09:02.255370 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:02.255344    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58553747-132d-4758-ad10-8df667f1c29d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e648ccdf-5x4w5\" (UID: \"58553747-132d-4758-ad10-8df667f1c29d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e648ccdf-5x4w5"
Oct 30 22:09:02.257017 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:02.256995    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58553747-132d-4758-ad10-8df667f1c29d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e648ccdf-5x4w5\" (UID: \"58553747-132d-4758-ad10-8df667f1c29d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e648ccdf-5x4w5"
Oct 30 22:09:02.268157 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:02.268138    5678 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-9vxbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Oct 30 22:09:02.268468 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:02.268178    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-9vxbk" podUID="bde7f877-4938-42d4-be6c-cba04795ce83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Oct 30 22:09:02.313623 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:02.313601    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2n858\" (UniqueName: \"kubernetes.io/projected/58553747-132d-4758-ad10-8df667f1c29d-kube-api-access-2n858\") pod \"maas-api-revoke-sub-e648ccdf-5x4w5\" (UID: \"58553747-132d-4758-ad10-8df667f1c29d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e648ccdf-5x4w5"
Oct 30 22:09:02.448476 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:02.446547    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e648ccdf-5x4w5"
Oct 30 22:09:02.835884 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:02.835841    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e648ccdf-5x4w5"]
Oct 30 22:09:02.837345 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:09:02.837317    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58553747_132d_4758_ad10_8df667f1c29d.slice/crio-faf7839df2ba224478bb93b43d532bd0a246ce7b47c44b5a62e1a34c6a4a8681 WatchSource:0}: Error finding container faf7839df2ba224478bb93b43d532bd0a246ce7b47c44b5a62e1a34c6a4a8681: Status 404 returned error can't find the container with id faf7839df2ba224478bb93b43d532bd0a246ce7b47c44b5a62e1a34c6a4a8681
Oct 30 22:09:03.269268 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:03.269234    5678 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-9vxbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Oct 30 22:09:03.269648 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:03.269291    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-9vxbk" podUID="bde7f877-4938-42d4-be6c-cba04795ce83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Oct 30 22:09:03.377626 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:03.377592    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e43442a0-9vt2m"]
Oct 30 22:09:03.380744 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:03.380693    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e43442a0-9vt2m"
Oct 30 22:09:03.443602 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:03.443575    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e43442a0-9vt2m"]
Oct 30 22:09:03.465588 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:03.465548    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e648ccdf-5x4w5" event={"ID":"58553747-132d-4758-ad10-8df667f1c29d","Type":"ContainerStarted","Data":"22c312f89e8803cbe93f399a895e0e852500657988c7d3f9f7a6de3c45987fd5"}
Oct 30 22:09:03.465745 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:03.465619    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e648ccdf-5x4w5" event={"ID":"58553747-132d-4758-ad10-8df667f1c29d","Type":"ContainerStarted","Data":"faf7839df2ba224478bb93b43d532bd0a246ce7b47c44b5a62e1a34c6a4a8681"}
Oct 30 22:09:03.469309 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:03.469282    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-955pj\" (UniqueName: \"kubernetes.io/projected/6a6f4173-cdfa-434d-9a78-a433495e76ed-kube-api-access-955pj\") pod \"maas-api-revoke-sub-e43442a0-9vt2m\" (UID: \"6a6f4173-cdfa-434d-9a78-a433495e76ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e43442a0-9vt2m"
Oct 30 22:09:03.469517 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:03.469495    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a6f4173-cdfa-434d-9a78-a433495e76ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e43442a0-9vt2m\" (UID: \"6a6f4173-cdfa-434d-9a78-a433495e76ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e43442a0-9vt2m"
Oct 30 22:09:03.469567 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:03.469533    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a6f4173-cdfa-434d-9a78-a433495e76ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e43442a0-9vt2m\" (UID: \"6a6f4173-cdfa-434d-9a78-a433495e76ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e43442a0-9vt2m"
Oct 30 22:09:03.570870 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:03.570842    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-955pj\" (UniqueName: \"kubernetes.io/projected/6a6f4173-cdfa-434d-9a78-a433495e76ed-kube-api-access-955pj\") pod \"maas-api-revoke-sub-e43442a0-9vt2m\" (UID: \"6a6f4173-cdfa-434d-9a78-a433495e76ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e43442a0-9vt2m"
Oct 30 22:09:03.571061 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:03.570994    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a6f4173-cdfa-434d-9a78-a433495e76ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e43442a0-9vt2m\" (UID: \"6a6f4173-cdfa-434d-9a78-a433495e76ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e43442a0-9vt2m"
Oct 30 22:09:03.571259 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:03.571215    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a6f4173-cdfa-434d-9a78-a433495e76ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e43442a0-9vt2m\" (UID: \"6a6f4173-cdfa-434d-9a78-a433495e76ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e43442a0-9vt2m"
Oct 30 22:09:03.571779 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:03.571752    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a6f4173-cdfa-434d-9a78-a433495e76ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e43442a0-9vt2m\" (UID: \"6a6f4173-cdfa-434d-9a78-a433495e76ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e43442a0-9vt2m"
Oct 30 22:09:03.573501 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:03.573478    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a6f4173-cdfa-434d-9a78-a433495e76ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e43442a0-9vt2m\" (UID: \"6a6f4173-cdfa-434d-9a78-a433495e76ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e43442a0-9vt2m"
Oct 30 22:09:03.606234 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:03.606191    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e648ccdf-5x4w5" podStartSLOduration=1.6061798010000001 podStartE2EDuration="1.606179801s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:09:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:09:03.602856334 +0000 UTC m=+1121.207163888" watchObservedRunningTime="2026-10-30 22:09:03.606179801 +0000 UTC m=+1121.210487405"
Oct 30 22:09:03.629613 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:03.629584    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-955pj\" (UniqueName: \"kubernetes.io/projected/6a6f4173-cdfa-434d-9a78-a433495e76ed-kube-api-access-955pj\") pod \"maas-api-revoke-sub-e43442a0-9vt2m\" (UID: \"6a6f4173-cdfa-434d-9a78-a433495e76ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e43442a0-9vt2m"
Oct 30 22:09:03.698548 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:03.698516    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e43442a0-9vt2m"
Oct 30 22:09:04.079669 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:04.079634    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e43442a0-9vt2m"]
Oct 30 22:09:04.080657 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:09:04.080624    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a6f4173_cdfa_434d_9a78_a433495e76ed.slice/crio-17d593fcf4a765e0b9c8fd50f91353ab0ef0a7ff75c47f65dcdfc4a3ffe20214 WatchSource:0}: Error finding container 17d593fcf4a765e0b9c8fd50f91353ab0ef0a7ff75c47f65dcdfc4a3ffe20214: Status 404 returned error can't find the container with id 17d593fcf4a765e0b9c8fd50f91353ab0ef0a7ff75c47f65dcdfc4a3ffe20214
Oct 30 22:09:04.268224 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:04.268188    5678 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-9vxbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Oct 30 22:09:04.268370 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:04.268248    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-9vxbk" podUID="bde7f877-4938-42d4-be6c-cba04795ce83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Oct 30 22:09:04.476970 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:04.476926    5678 generic.go:356] "Generic (PLEG): container finished" podID="58553747-132d-4758-ad10-8df667f1c29d" containerID="22c312f89e8803cbe93f399a895e0e852500657988c7d3f9f7a6de3c45987fd5" exitCode=0
Oct 30 22:09:04.477388 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:04.476976    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e648ccdf-5x4w5" event={"ID":"58553747-132d-4758-ad10-8df667f1c29d","Type":"ContainerDied","Data":"22c312f89e8803cbe93f399a895e0e852500657988c7d3f9f7a6de3c45987fd5"}
Oct 30 22:09:04.481421 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:04.478992    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e43442a0-9vt2m" event={"ID":"6a6f4173-cdfa-434d-9a78-a433495e76ed","Type":"ContainerStarted","Data":"85191fe5eb85b0cb770344671e29fa69f048487d18093e591872e1f6e2193200"}
Oct 30 22:09:04.481421 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:04.479039    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e43442a0-9vt2m" event={"ID":"6a6f4173-cdfa-434d-9a78-a433495e76ed","Type":"ContainerStarted","Data":"17d593fcf4a765e0b9c8fd50f91353ab0ef0a7ff75c47f65dcdfc4a3ffe20214"}
Oct 30 22:09:04.705076 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:04.705025    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e43442a0-9vt2m" podStartSLOduration=1.705011803 podStartE2EDuration="1.705011803s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:09:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:09:04.703913174 +0000 UTC m=+1122.308220730" watchObservedRunningTime="2026-10-30 22:09:04.705011803 +0000 UTC m=+1122.309319356"
Oct 30 22:09:05.272597 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:05.272564    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-9vxbk"
Oct 30 22:09:05.273152 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:05.273042    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-9vxbk"
Oct 30 22:09:05.274249 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:05.274228    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-9vxbk"
Oct 30 22:09:05.487693 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:05.487657    5678 generic.go:356] "Generic (PLEG): container finished" podID="6a6f4173-cdfa-434d-9a78-a433495e76ed" containerID="85191fe5eb85b0cb770344671e29fa69f048487d18093e591872e1f6e2193200" exitCode=0
Oct 30 22:09:05.488090 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:05.487738    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e43442a0-9vt2m" event={"ID":"6a6f4173-cdfa-434d-9a78-a433495e76ed","Type":"ContainerDied","Data":"85191fe5eb85b0cb770344671e29fa69f048487d18093e591872e1f6e2193200"}
Oct 30 22:09:05.676638 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:05.676601    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e648ccdf-5x4w5"
Oct 30 22:09:05.693555 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:05.693529    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58553747-132d-4758-ad10-8df667f1c29d-kube-api-access-2n858\" (UniqueName: \"kubernetes.io/projected/58553747-132d-4758-ad10-8df667f1c29d-kube-api-access-2n858\") pod \"58553747-132d-4758-ad10-8df667f1c29d\" (UID: \"58553747-132d-4758-ad10-8df667f1c29d\") "
Oct 30 22:09:05.693738 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:05.693616    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58553747-132d-4758-ad10-8df667f1c29d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58553747-132d-4758-ad10-8df667f1c29d-maas-api-openshift-ca\") pod \"58553747-132d-4758-ad10-8df667f1c29d\" (UID: \"58553747-132d-4758-ad10-8df667f1c29d\") "
Oct 30 22:09:05.693738 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:05.693681    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58553747-132d-4758-ad10-8df667f1c29d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58553747-132d-4758-ad10-8df667f1c29d-maas-api-xks-ca\") pod \"58553747-132d-4758-ad10-8df667f1c29d\" (UID: \"58553747-132d-4758-ad10-8df667f1c29d\") "
Oct 30 22:09:05.694014 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:05.693986    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58553747-132d-4758-ad10-8df667f1c29d-maas-api-openshift-ca" pod "58553747-132d-4758-ad10-8df667f1c29d" (UID: "58553747-132d-4758-ad10-8df667f1c29d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:09:05.694174 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:05.694151    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58553747-132d-4758-ad10-8df667f1c29d-maas-api-openshift-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:09:05.695704 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:05.695677    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58553747-132d-4758-ad10-8df667f1c29d-kube-api-access-2n858" pod "58553747-132d-4758-ad10-8df667f1c29d" (UID: "58553747-132d-4758-ad10-8df667f1c29d"). InnerVolumeSpecName "kube-api-access-2n858". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:09:05.695819 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:05.695793    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58553747-132d-4758-ad10-8df667f1c29d-maas-api-xks-ca" pod "58553747-132d-4758-ad10-8df667f1c29d" (UID: "58553747-132d-4758-ad10-8df667f1c29d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:09:05.795642 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:05.795607    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58553747-132d-4758-ad10-8df667f1c29d-maas-api-xks-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:09:05.795642 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:05.795642    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2n858\" (UniqueName: \"kubernetes.io/projected/58553747-132d-4758-ad10-8df667f1c29d-kube-api-access-2n858\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:09:06.495619 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:06.495585    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e648ccdf-5x4w5" event={"ID":"58553747-132d-4758-ad10-8df667f1c29d","Type":"ContainerDied","Data":"faf7839df2ba224478bb93b43d532bd0a246ce7b47c44b5a62e1a34c6a4a8681"}
Oct 30 22:09:06.496102 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:06.495634    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e648ccdf-5x4w5"
Oct 30 22:09:06.496102 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:06.495668    5678 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="faf7839df2ba224478bb93b43d532bd0a246ce7b47c44b5a62e1a34c6a4a8681"
Oct 30 22:09:06.650423 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:06.650384    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e43442a0-9vt2m"
Oct 30 22:09:06.705121 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:06.705088    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a6f4173-cdfa-434d-9a78-a433495e76ed-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a6f4173-cdfa-434d-9a78-a433495e76ed-maas-api-openshift-ca\") pod \"6a6f4173-cdfa-434d-9a78-a433495e76ed\" (UID: \"6a6f4173-cdfa-434d-9a78-a433495e76ed\") "
Oct 30 22:09:06.705278 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:06.705260    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a6f4173-cdfa-434d-9a78-a433495e76ed-kube-api-access-955pj\" (UniqueName: \"kubernetes.io/projected/6a6f4173-cdfa-434d-9a78-a433495e76ed-kube-api-access-955pj\") pod \"6a6f4173-cdfa-434d-9a78-a433495e76ed\" (UID: \"6a6f4173-cdfa-434d-9a78-a433495e76ed\") "
Oct 30 22:09:06.705320 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:06.705300    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a6f4173-cdfa-434d-9a78-a433495e76ed-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a6f4173-cdfa-434d-9a78-a433495e76ed-maas-api-xks-ca\") pod \"6a6f4173-cdfa-434d-9a78-a433495e76ed\" (UID: \"6a6f4173-cdfa-434d-9a78-a433495e76ed\") "
Oct 30 22:09:06.705500 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:06.705478    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a6f4173-cdfa-434d-9a78-a433495e76ed-maas-api-openshift-ca" pod "6a6f4173-cdfa-434d-9a78-a433495e76ed" (UID: "6a6f4173-cdfa-434d-9a78-a433495e76ed"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:09:06.705790 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:06.705767    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a6f4173-cdfa-434d-9a78-a433495e76ed-maas-api-openshift-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:09:06.708619 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:06.708590    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a6f4173-cdfa-434d-9a78-a433495e76ed-maas-api-xks-ca" pod "6a6f4173-cdfa-434d-9a78-a433495e76ed" (UID: "6a6f4173-cdfa-434d-9a78-a433495e76ed"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:09:06.708619 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:06.708604    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a6f4173-cdfa-434d-9a78-a433495e76ed-kube-api-access-955pj" pod "6a6f4173-cdfa-434d-9a78-a433495e76ed" (UID: "6a6f4173-cdfa-434d-9a78-a433495e76ed"). InnerVolumeSpecName "kube-api-access-955pj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:09:06.807386 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:06.807311    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-955pj\" (UniqueName: \"kubernetes.io/projected/6a6f4173-cdfa-434d-9a78-a433495e76ed-kube-api-access-955pj\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:09:06.807386 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:06.807339    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a6f4173-cdfa-434d-9a78-a433495e76ed-maas-api-xks-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:09:07.404824 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:07.404785    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81e6c9d9-kbhlz"]
Oct 30 22:09:07.405460 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:07.405437    5678 cpu_manager.go:414] "containerMap: removing container" podUID="58553747-132d-4758-ad10-8df667f1c29d" containerName="revoke-keys"
Oct 30 22:09:07.405460 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:07.405460    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58553747-132d-4758-ad10-8df667f1c29d" containerName="revoke-keys"
Oct 30 22:09:07.405612 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:07.405525    5678 cpu_manager.go:414] "containerMap: removing container" podUID="6a6f4173-cdfa-434d-9a78-a433495e76ed" containerName="revoke-keys"
Oct 30 22:09:07.405612 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:07.405533    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a6f4173-cdfa-434d-9a78-a433495e76ed" containerName="revoke-keys"
Oct 30 22:09:07.405721 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:07.405701    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a6f4173-cdfa-434d-9a78-a433495e76ed" containerName="revoke-keys"
Oct 30 22:09:07.405721 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:07.405722    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="58553747-132d-4758-ad10-8df667f1c29d" containerName="revoke-keys"
Oct 30 22:09:07.408987 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:07.408969    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81e6c9d9-kbhlz"
Oct 30 22:09:07.463803 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:07.463769    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81e6c9d9-kbhlz"]
Oct 30 22:09:07.503561 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:07.503529    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e43442a0-9vt2m"
Oct 30 22:09:07.504039 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:07.503525    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e43442a0-9vt2m" event={"ID":"6a6f4173-cdfa-434d-9a78-a433495e76ed","Type":"ContainerDied","Data":"17d593fcf4a765e0b9c8fd50f91353ab0ef0a7ff75c47f65dcdfc4a3ffe20214"}
Oct 30 22:09:07.504039 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:07.503657    5678 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="17d593fcf4a765e0b9c8fd50f91353ab0ef0a7ff75c47f65dcdfc4a3ffe20214"
Oct 30 22:09:07.514455 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:07.514431    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8de2e4f8-e41f-466f-b376-6408c6ac5d65-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81e6c9d9-kbhlz\" (UID: \"8de2e4f8-e41f-466f-b376-6408c6ac5d65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81e6c9d9-kbhlz"
Oct 30 22:09:07.514558 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:07.514474    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68r4x\" (UniqueName: \"kubernetes.io/projected/8de2e4f8-e41f-466f-b376-6408c6ac5d65-kube-api-access-68r4x\") pod \"maas-api-revoke-sub-81e6c9d9-kbhlz\" (UID: \"8de2e4f8-e41f-466f-b376-6408c6ac5d65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81e6c9d9-kbhlz"
Oct 30 22:09:07.514686 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:07.514672    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8de2e4f8-e41f-466f-b376-6408c6ac5d65-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81e6c9d9-kbhlz\" (UID: \"8de2e4f8-e41f-466f-b376-6408c6ac5d65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81e6c9d9-kbhlz"
Oct 30 22:09:07.615679 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:07.615650    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8de2e4f8-e41f-466f-b376-6408c6ac5d65-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81e6c9d9-kbhlz\" (UID: \"8de2e4f8-e41f-466f-b376-6408c6ac5d65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81e6c9d9-kbhlz"
Oct 30 22:09:07.615840 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:07.615695    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8de2e4f8-e41f-466f-b376-6408c6ac5d65-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81e6c9d9-kbhlz\" (UID: \"8de2e4f8-e41f-466f-b376-6408c6ac5d65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81e6c9d9-kbhlz"
Oct 30 22:09:07.615840 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:07.615723    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68r4x\" (UniqueName: \"kubernetes.io/projected/8de2e4f8-e41f-466f-b376-6408c6ac5d65-kube-api-access-68r4x\") pod \"maas-api-revoke-sub-81e6c9d9-kbhlz\" (UID: \"8de2e4f8-e41f-466f-b376-6408c6ac5d65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81e6c9d9-kbhlz"
Oct 30 22:09:07.616361 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:07.616341    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8de2e4f8-e41f-466f-b376-6408c6ac5d65-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81e6c9d9-kbhlz\" (UID: \"8de2e4f8-e41f-466f-b376-6408c6ac5d65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81e6c9d9-kbhlz"
Oct 30 22:09:07.618117 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:07.618084    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8de2e4f8-e41f-466f-b376-6408c6ac5d65-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81e6c9d9-kbhlz\" (UID: \"8de2e4f8-e41f-466f-b376-6408c6ac5d65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81e6c9d9-kbhlz"
Oct 30 22:09:07.676853 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:07.676798    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68r4x\" (UniqueName: \"kubernetes.io/projected/8de2e4f8-e41f-466f-b376-6408c6ac5d65-kube-api-access-68r4x\") pod \"maas-api-revoke-sub-81e6c9d9-kbhlz\" (UID: \"8de2e4f8-e41f-466f-b376-6408c6ac5d65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81e6c9d9-kbhlz"
Oct 30 22:09:07.722290 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:07.722265    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81e6c9d9-kbhlz"
Oct 30 22:09:08.102583 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:08.102528    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81e6c9d9-kbhlz"]
Oct 30 22:09:08.102925 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:09:08.102893    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8de2e4f8_e41f_466f_b376_6408c6ac5d65.slice/crio-0b33866c896c066289a1d4a19c3a4471b7bb175085f06a10ec469172f9ec34fe WatchSource:0}: Error finding container 0b33866c896c066289a1d4a19c3a4471b7bb175085f06a10ec469172f9ec34fe: Status 404 returned error can't find the container with id 0b33866c896c066289a1d4a19c3a4471b7bb175085f06a10ec469172f9ec34fe
Oct 30 22:09:08.539210 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:08.539173    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81e6c9d9-kbhlz" event={"ID":"8de2e4f8-e41f-466f-b376-6408c6ac5d65","Type":"ContainerStarted","Data":"3de1003a8b4a38f0b001bb41db04fc0fd22e58b894f1b657c8cc060c7385ea59"}
Oct 30 22:09:08.539657 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:08.539231    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81e6c9d9-kbhlz" event={"ID":"8de2e4f8-e41f-466f-b376-6408c6ac5d65","Type":"ContainerStarted","Data":"0b33866c896c066289a1d4a19c3a4471b7bb175085f06a10ec469172f9ec34fe"}
Oct 30 22:09:08.639593 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:08.639539    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81e6c9d9-kbhlz" podStartSLOduration=1.639526789 podStartE2EDuration="1.639526789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:09:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:09:08.637539117 +0000 UTC m=+1126.241846694" watchObservedRunningTime="2026-10-30 22:09:08.639526789 +0000 UTC m=+1126.243834373"
Oct 30 22:09:09.533179 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:09.533144    5678 generic.go:356] "Generic (PLEG): container finished" podID="8de2e4f8-e41f-466f-b376-6408c6ac5d65" containerID="3de1003a8b4a38f0b001bb41db04fc0fd22e58b894f1b657c8cc060c7385ea59" exitCode=0
Oct 30 22:09:09.533349 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:09.533226    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81e6c9d9-kbhlz" event={"ID":"8de2e4f8-e41f-466f-b376-6408c6ac5d65","Type":"ContainerDied","Data":"3de1003a8b4a38f0b001bb41db04fc0fd22e58b894f1b657c8cc060c7385ea59"}
Oct 30 22:09:10.572026 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:09:10.571984    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 22:09:10.689446 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:10.689389    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81e6c9d9-kbhlz"
Oct 30 22:09:10.750733 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:10.750705    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8de2e4f8-e41f-466f-b376-6408c6ac5d65-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8de2e4f8-e41f-466f-b376-6408c6ac5d65-maas-api-xks-ca\") pod \"8de2e4f8-e41f-466f-b376-6408c6ac5d65\" (UID: \"8de2e4f8-e41f-466f-b376-6408c6ac5d65\") "
Oct 30 22:09:10.750911 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:10.750779    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8de2e4f8-e41f-466f-b376-6408c6ac5d65-kube-api-access-68r4x\" (UniqueName: \"kubernetes.io/projected/8de2e4f8-e41f-466f-b376-6408c6ac5d65-kube-api-access-68r4x\") pod \"8de2e4f8-e41f-466f-b376-6408c6ac5d65\" (UID: \"8de2e4f8-e41f-466f-b376-6408c6ac5d65\") "
Oct 30 22:09:10.750911 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:10.750877    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8de2e4f8-e41f-466f-b376-6408c6ac5d65-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8de2e4f8-e41f-466f-b376-6408c6ac5d65-maas-api-openshift-ca\") pod \"8de2e4f8-e41f-466f-b376-6408c6ac5d65\" (UID: \"8de2e4f8-e41f-466f-b376-6408c6ac5d65\") "
Oct 30 22:09:10.751223 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:10.751196    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8de2e4f8-e41f-466f-b376-6408c6ac5d65-maas-api-openshift-ca" pod "8de2e4f8-e41f-466f-b376-6408c6ac5d65" (UID: "8de2e4f8-e41f-466f-b376-6408c6ac5d65"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:09:10.752857 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:10.752833    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8de2e4f8-e41f-466f-b376-6408c6ac5d65-kube-api-access-68r4x" pod "8de2e4f8-e41f-466f-b376-6408c6ac5d65" (UID: "8de2e4f8-e41f-466f-b376-6408c6ac5d65"). InnerVolumeSpecName "kube-api-access-68r4x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:09:10.752954 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:10.752896    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8de2e4f8-e41f-466f-b376-6408c6ac5d65-maas-api-xks-ca" pod "8de2e4f8-e41f-466f-b376-6408c6ac5d65" (UID: "8de2e4f8-e41f-466f-b376-6408c6ac5d65"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:09:10.852060 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:10.851982    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8de2e4f8-e41f-466f-b376-6408c6ac5d65-maas-api-xks-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:09:10.852060 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:10.852006    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68r4x\" (UniqueName: \"kubernetes.io/projected/8de2e4f8-e41f-466f-b376-6408c6ac5d65-kube-api-access-68r4x\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:09:10.852060 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:10.852026    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8de2e4f8-e41f-466f-b376-6408c6ac5d65-maas-api-openshift-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:09:11.547679 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:11.547647    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81e6c9d9-kbhlz"
Oct 30 22:09:11.547910 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:11.547688    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81e6c9d9-kbhlz" event={"ID":"8de2e4f8-e41f-466f-b376-6408c6ac5d65","Type":"ContainerDied","Data":"0b33866c896c066289a1d4a19c3a4471b7bb175085f06a10ec469172f9ec34fe"}
Oct 30 22:09:11.547910 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:11.547728    5678 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0b33866c896c066289a1d4a19c3a4471b7bb175085f06a10ec469172f9ec34fe"
Oct 30 22:09:12.925171 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:12.925140    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00d9b33b-tcsgs"]
Oct 30 22:09:12.925803 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:12.925781    5678 cpu_manager.go:414] "containerMap: removing container" podUID="8de2e4f8-e41f-466f-b376-6408c6ac5d65" containerName="revoke-keys"
Oct 30 22:09:12.925855 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:12.925806    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8de2e4f8-e41f-466f-b376-6408c6ac5d65" containerName="revoke-keys"
Oct 30 22:09:12.925995 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:12.925982    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="8de2e4f8-e41f-466f-b376-6408c6ac5d65" containerName="revoke-keys"
Oct 30 22:09:12.929441 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:12.929422    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00d9b33b-tcsgs"
Oct 30 22:09:12.976073 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:12.976044    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61aa9c47-8147-4d4e-852d-0973ba1c5c01-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00d9b33b-tcsgs\" (UID: \"61aa9c47-8147-4d4e-852d-0973ba1c5c01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00d9b33b-tcsgs"
Oct 30 22:09:12.976293 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:12.976260    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61aa9c47-8147-4d4e-852d-0973ba1c5c01-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00d9b33b-tcsgs\" (UID: \"61aa9c47-8147-4d4e-852d-0973ba1c5c01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00d9b33b-tcsgs"
Oct 30 22:09:12.976436 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:12.976307    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glg82\" (UniqueName: \"kubernetes.io/projected/61aa9c47-8147-4d4e-852d-0973ba1c5c01-kube-api-access-glg82\") pod \"maas-api-revoke-sub-00d9b33b-tcsgs\" (UID: \"61aa9c47-8147-4d4e-852d-0973ba1c5c01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00d9b33b-tcsgs"
Oct 30 22:09:12.984647 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:12.984621    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00d9b33b-tcsgs"]
Oct 30 22:09:13.026428 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:13.026392    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:09:13.026612 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:13.026454    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5rvcj\""
Oct 30 22:09:13.077887 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:13.077855    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61aa9c47-8147-4d4e-852d-0973ba1c5c01-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00d9b33b-tcsgs\" (UID: \"61aa9c47-8147-4d4e-852d-0973ba1c5c01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00d9b33b-tcsgs"
Oct 30 22:09:13.078052 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:13.078028    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61aa9c47-8147-4d4e-852d-0973ba1c5c01-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00d9b33b-tcsgs\" (UID: \"61aa9c47-8147-4d4e-852d-0973ba1c5c01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00d9b33b-tcsgs"
Oct 30 22:09:13.078115 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:13.078059    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glg82\" (UniqueName: \"kubernetes.io/projected/61aa9c47-8147-4d4e-852d-0973ba1c5c01-kube-api-access-glg82\") pod \"maas-api-revoke-sub-00d9b33b-tcsgs\" (UID: \"61aa9c47-8147-4d4e-852d-0973ba1c5c01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00d9b33b-tcsgs"
Oct 30 22:09:13.078630 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:13.078607    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61aa9c47-8147-4d4e-852d-0973ba1c5c01-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00d9b33b-tcsgs\" (UID: \"61aa9c47-8147-4d4e-852d-0973ba1c5c01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00d9b33b-tcsgs"
Oct 30 22:09:13.080075 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:13.080056    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61aa9c47-8147-4d4e-852d-0973ba1c5c01-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00d9b33b-tcsgs\" (UID: \"61aa9c47-8147-4d4e-852d-0973ba1c5c01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00d9b33b-tcsgs"
Oct 30 22:09:13.136464 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:13.136435    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-glg82\" (UniqueName: \"kubernetes.io/projected/61aa9c47-8147-4d4e-852d-0973ba1c5c01-kube-api-access-glg82\") pod \"maas-api-revoke-sub-00d9b33b-tcsgs\" (UID: \"61aa9c47-8147-4d4e-852d-0973ba1c5c01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00d9b33b-tcsgs"
Oct 30 22:09:13.244383 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:13.244297    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00d9b33b-tcsgs"
Oct 30 22:09:13.626338 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:13.626302    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00d9b33b-tcsgs"]
Oct 30 22:09:13.627518 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:09:13.627492    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61aa9c47_8147_4d4e_852d_0973ba1c5c01.slice/crio-382c6698e507c1f8a14f514a485245f721bc0ce5c0ff38f847477f667890ec15 WatchSource:0}: Error finding container 382c6698e507c1f8a14f514a485245f721bc0ce5c0ff38f847477f667890ec15: Status 404 returned error can't find the container with id 382c6698e507c1f8a14f514a485245f721bc0ce5c0ff38f847477f667890ec15
Oct 30 22:09:14.570943 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:14.570908    5678 generic.go:356] "Generic (PLEG): container finished" podID="61aa9c47-8147-4d4e-852d-0973ba1c5c01" containerID="dcc3ab333b217e67f6cb49a6c30b25b1f9ba35abbc667182318f0ac1d27b43f7" exitCode=0
Oct 30 22:09:14.571376 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:14.570939    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00d9b33b-tcsgs" event={"ID":"61aa9c47-8147-4d4e-852d-0973ba1c5c01","Type":"ContainerDied","Data":"dcc3ab333b217e67f6cb49a6c30b25b1f9ba35abbc667182318f0ac1d27b43f7"}
Oct 30 22:09:14.571376 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:14.570989    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00d9b33b-tcsgs" event={"ID":"61aa9c47-8147-4d4e-852d-0973ba1c5c01","Type":"ContainerStarted","Data":"382c6698e507c1f8a14f514a485245f721bc0ce5c0ff38f847477f667890ec15"}
Oct 30 22:09:16.210246 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:16.210201    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-jps22"]
Oct 30 22:09:16.225956 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:16.225729    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-jps22"]
Oct 30 22:09:16.225956 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:16.225857    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-jps22"
Oct 30 22:09:16.316616 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:16.316584    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00d9b33b-tcsgs"
Oct 30 22:09:16.391096 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:16.391048    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d167328f-3684-41d7-9407-5ceec39dd03f-workload-certs\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-jps22\" (UID: \"d167328f-3684-41d7-9407-5ceec39dd03f\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-jps22"
Oct 30 22:09:16.391306 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:16.391225    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d167328f-3684-41d7-9407-5ceec39dd03f-istio-token\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-jps22\" (UID: \"d167328f-3684-41d7-9407-5ceec39dd03f\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-jps22"
Oct 30 22:09:16.391306 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:16.391274    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d167328f-3684-41d7-9407-5ceec39dd03f-istio-podinfo\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-jps22\" (UID: \"d167328f-3684-41d7-9407-5ceec39dd03f\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-jps22"
Oct 30 22:09:16.391446 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:16.391338    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d167328f-3684-41d7-9407-5ceec39dd03f-workload-socket\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-jps22\" (UID: \"d167328f-3684-41d7-9407-5ceec39dd03f\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-jps22"
Oct 30 22:09:16.391446 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:16.391394    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d167328f-3684-41d7-9407-5ceec39dd03f-credential-socket\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-jps22\" (UID: \"d167328f-3684-41d7-9407-5ceec39dd03f\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-jps22"
Oct 30 22:09:16.391520 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:16.391484    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d167328f-3684-41d7-9407-5ceec39dd03f-istiod-ca-cert\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-jps22\" (UID: \"d167328f-3684-41d7-9407-5ceec39dd03f\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-jps22"
Oct 30 22:09:16.391520 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:16.391509    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d167328f-3684-41d7-9407-5ceec39dd03f-istio-data\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-jps22\" (UID: \"d167328f-3684-41d7-9407-5ceec39dd03f\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-jps22"
Oct 30 22:09:16.391585 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:16.391542    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d167328f-3684-41d7-9407-5ceec39dd03f-istio-envoy\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-jps22\" (UID: \"d167328f-3684-41d7-9407-5ceec39dd03f\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-jps22"
Oct 30 22:09:16.391674 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:16.391653    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ncwx\" (UniqueName: \"kubernetes.io/projected/d167328f-3684-41d7-9407-5ceec39dd03f-kube-api-access-7ncwx\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-jps22\" (UID: \"d167328f-3684-41d7-9407-5ceec39dd03f\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-jps22"
Oct 30 22:09:16.493597 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:16.493558    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61aa9c47-8147-4d4e-852d-0973ba1c5c01-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61aa9c47-8147-4d4e-852d-0973ba1c5c01-maas-api-openshift-ca\") pod \"61aa9c47-8147-4d4e-852d-0973ba1c5c01\" (UID: \"61aa9c47-8147-4d4e-852d-0973ba1c5c01\") "
Oct 30 22:09:16.493774 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:16.493658    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61aa9c47-8147-4d4e-852d-0973ba1c5c01-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61aa9c47-8147-4d4e-852d-0973ba1c5c01-maas-api-xks-ca\") pod \"61aa9c47-8147-4d4e-852d-0973ba1c5c01\" (UID: \"61aa9c47-8147-4d4e-852d-0973ba1c5c01\") "
Oct 30 22:09:16.493774 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:16.493764    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61aa9c47-8147-4d4e-852d-0973ba1c5c01-kube-api-access-glg82\" (UniqueName: \"kubernetes.io/projected/61aa9c47-8147-4d4e-852d-0973ba1c5c01-kube-api-access-glg82\") pod \"61aa9c47-8147-4d4e-852d-0973ba1c5c01\" (UID: \"61aa9c47-8147-4d4e-852d-0973ba1c5c01\") "
Oct 30 22:09:16.494058 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:16.494037    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d167328f-3684-41d7-9407-5ceec39dd03f-istio-token\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-jps22\" (UID: \"d167328f-3684-41d7-9407-5ceec39dd03f\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-jps22"
Oct 30 22:09:16.494177 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:16.494093    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d167328f-3684-41d7-9407-5ceec39dd03f-istio-podinfo\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-jps22\" (UID: \"d167328f-3684-41d7-9407-5ceec39dd03f\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-jps22"
Oct 30 22:09:16.494177 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:16.494122    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d167328f-3684-41d7-9407-5ceec39dd03f-workload-socket\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-jps22\" (UID: \"d167328f-3684-41d7-9407-5ceec39dd03f\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-jps22"
Oct 30 22:09:16.494177 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:16.494145    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d167328f-3684-41d7-9407-5ceec39dd03f-credential-socket\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-jps22\" (UID: \"d167328f-3684-41d7-9407-5ceec39dd03f\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-jps22"
Oct 30 22:09:16.494338 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:16.494189    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d167328f-3684-41d7-9407-5ceec39dd03f-istiod-ca-cert\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-jps22\" (UID: \"d167328f-3684-41d7-9407-5ceec39dd03f\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-jps22"
Oct 30 22:09:16.494338 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:16.494225    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d167328f-3684-41d7-9407-5ceec39dd03f-istio-data\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-jps22\" (UID: \"d167328f-3684-41d7-9407-5ceec39dd03f\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-jps22"
Oct 30 22:09:16.494338 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:16.494278    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d167328f-3684-41d7-9407-5ceec39dd03f-istio-envoy\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-jps22\" (UID: \"d167328f-3684-41d7-9407-5ceec39dd03f\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-jps22"
Oct 30 22:09:16.494516 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:16.494344    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ncwx\" (UniqueName: \"kubernetes.io/projected/d167328f-3684-41d7-9407-5ceec39dd03f-kube-api-access-7ncwx\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-jps22\" (UID: \"d167328f-3684-41d7-9407-5ceec39dd03f\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-jps22"
Oct 30 22:09:16.494516 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:16.494434    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d167328f-3684-41d7-9407-5ceec39dd03f-workload-certs\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-jps22\" (UID: \"d167328f-3684-41d7-9407-5ceec39dd03f\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-jps22"
Oct 30 22:09:16.494998 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:16.494815    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d167328f-3684-41d7-9407-5ceec39dd03f-workload-certs\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-jps22\" (UID: \"d167328f-3684-41d7-9407-5ceec39dd03f\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-jps22"
Oct 30 22:09:16.495202 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:16.495179    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61aa9c47-8147-4d4e-852d-0973ba1c5c01-maas-api-openshift-ca" pod "61aa9c47-8147-4d4e-852d-0973ba1c5c01" (UID: "61aa9c47-8147-4d4e-852d-0973ba1c5c01"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:09:16.496733 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:16.496704    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d167328f-3684-41d7-9407-5ceec39dd03f-credential-socket\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-jps22\" (UID: \"d167328f-3684-41d7-9407-5ceec39dd03f\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-jps22"
Oct 30 22:09:16.497787 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:16.497758    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d167328f-3684-41d7-9407-5ceec39dd03f-workload-socket\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-jps22\" (UID: \"d167328f-3684-41d7-9407-5ceec39dd03f\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-jps22"
Oct 30 22:09:16.498159 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:16.498132    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d167328f-3684-41d7-9407-5ceec39dd03f-istio-data\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-jps22\" (UID: \"d167328f-3684-41d7-9407-5ceec39dd03f\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-jps22"
Oct 30 22:09:16.498354 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:16.498333    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d167328f-3684-41d7-9407-5ceec39dd03f-istiod-ca-cert\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-jps22\" (UID: \"d167328f-3684-41d7-9407-5ceec39dd03f\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-jps22"
Oct 30 22:09:16.499478 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:16.499003    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61aa9c47-8147-4d4e-852d-0973ba1c5c01-maas-api-xks-ca" pod "61aa9c47-8147-4d4e-852d-0973ba1c5c01" (UID: "61aa9c47-8147-4d4e-852d-0973ba1c5c01"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:09:16.505950 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:16.505898    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61aa9c47-8147-4d4e-852d-0973ba1c5c01-kube-api-access-glg82" pod "61aa9c47-8147-4d4e-852d-0973ba1c5c01" (UID: "61aa9c47-8147-4d4e-852d-0973ba1c5c01"). InnerVolumeSpecName "kube-api-access-glg82". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:09:16.505950 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:16.505947    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d167328f-3684-41d7-9407-5ceec39dd03f-istio-envoy\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-jps22\" (UID: \"d167328f-3684-41d7-9407-5ceec39dd03f\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-jps22"
Oct 30 22:09:16.507827 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:16.507808    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d167328f-3684-41d7-9407-5ceec39dd03f-istio-podinfo\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-jps22\" (UID: \"d167328f-3684-41d7-9407-5ceec39dd03f\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-jps22"
Oct 30 22:09:16.570176 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:16.560799    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ncwx\" (UniqueName: \"kubernetes.io/projected/d167328f-3684-41d7-9407-5ceec39dd03f-kube-api-access-7ncwx\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-jps22\" (UID: \"d167328f-3684-41d7-9407-5ceec39dd03f\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-jps22"
Oct 30 22:09:16.570176 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:16.566551    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d167328f-3684-41d7-9407-5ceec39dd03f-istio-token\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-jps22\" (UID: \"d167328f-3684-41d7-9407-5ceec39dd03f\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-jps22"
Oct 30 22:09:16.595720 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:16.595683    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61aa9c47-8147-4d4e-852d-0973ba1c5c01-maas-api-openshift-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:09:16.595862 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:16.595735    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61aa9c47-8147-4d4e-852d-0973ba1c5c01-maas-api-xks-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:09:16.595862 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:16.595760    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glg82\" (UniqueName: \"kubernetes.io/projected/61aa9c47-8147-4d4e-852d-0973ba1c5c01-kube-api-access-glg82\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:09:16.678453 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:16.678416    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00d9b33b-tcsgs" event={"ID":"61aa9c47-8147-4d4e-852d-0973ba1c5c01","Type":"ContainerDied","Data":"382c6698e507c1f8a14f514a485245f721bc0ce5c0ff38f847477f667890ec15"}
Oct 30 22:09:16.678626 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:16.678484    5678 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="382c6698e507c1f8a14f514a485245f721bc0ce5c0ff38f847477f667890ec15"
Oct 30 22:09:16.678626 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:16.678428    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00d9b33b-tcsgs"
Oct 30 22:09:16.849645 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:16.849566    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-jps22"
Oct 30 22:09:17.296335 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:17.296306    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-jps22"]
Oct 30 22:09:17.296960 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:09:17.296931    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd167328f_3684_41d7_9407_5ceec39dd03f.slice/crio-fbc41e71ada7332d6a73aa54eeef55bd72d6b4e2c300c66d20dba2e295b3d667 WatchSource:0}: Error finding container fbc41e71ada7332d6a73aa54eeef55bd72d6b4e2c300c66d20dba2e295b3d667: Status 404 returned error can't find the container with id fbc41e71ada7332d6a73aa54eeef55bd72d6b4e2c300c66d20dba2e295b3d667
Oct 30 22:09:17.299810 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:17.299778    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:09:17.299917 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:17.299861    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:09:17.299917 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:17.299907    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:09:17.686815 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:17.686777    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-jps22" event={"ID":"d167328f-3684-41d7-9407-5ceec39dd03f","Type":"ContainerStarted","Data":"f37eeb0872bd53abc956a7e3019a5df01eb46cfff5dbaa91d136ab47473f9105"}
Oct 30 22:09:17.686984 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:17.686840    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-jps22" event={"ID":"d167328f-3684-41d7-9407-5ceec39dd03f","Type":"ContainerStarted","Data":"fbc41e71ada7332d6a73aa54eeef55bd72d6b4e2c300c66d20dba2e295b3d667"}
Oct 30 22:09:17.806920 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:17.806870    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-jps22" podStartSLOduration=1.806856681 podStartE2EDuration="1.806856681s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:09:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:09:17.805358257 +0000 UTC m=+1135.409665842" watchObservedRunningTime="2026-10-30 22:09:17.806856681 +0000 UTC m=+1135.411164236"
Oct 30 22:09:17.849984 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:17.849949    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-jps22"
Oct 30 22:09:18.852752 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:18.852716    5678 patch_prober.go:29] interesting pod/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-jps22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Oct 30 22:09:18.853144 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:18.852770    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-jps22" podUID="d167328f-3684-41d7-9407-5ceec39dd03f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Oct 30 22:09:19.752182 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:19.752148    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-344071f5-wnlx2"]
Oct 30 22:09:19.752749 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:19.752730    5678 cpu_manager.go:414] "containerMap: removing container" podUID="61aa9c47-8147-4d4e-852d-0973ba1c5c01" containerName="revoke-keys"
Oct 30 22:09:19.752749 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:19.752747    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61aa9c47-8147-4d4e-852d-0973ba1c5c01" containerName="revoke-keys"
Oct 30 22:09:19.752943 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:19.752858    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="61aa9c47-8147-4d4e-852d-0973ba1c5c01" containerName="revoke-keys"
Oct 30 22:09:19.756453 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:19.756435    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-344071f5-wnlx2"
Oct 30 22:09:19.763196 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:19.763170    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5191ae0-55f2-4180-b01e-226183a6a225-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-344071f5-wnlx2\" (UID: \"c5191ae0-55f2-4180-b01e-226183a6a225\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-344071f5-wnlx2"
Oct 30 22:09:19.763293 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:19.763206    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbb6n\" (UniqueName: \"kubernetes.io/projected/c5191ae0-55f2-4180-b01e-226183a6a225-kube-api-access-pbb6n\") pod \"maas-api-revoke-sub-344071f5-wnlx2\" (UID: \"c5191ae0-55f2-4180-b01e-226183a6a225\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-344071f5-wnlx2"
Oct 30 22:09:19.763293 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:19.763231    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5191ae0-55f2-4180-b01e-226183a6a225-maas-api-xks-ca\") pod \"maas-api-revoke-sub-344071f5-wnlx2\" (UID: \"c5191ae0-55f2-4180-b01e-226183a6a225\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-344071f5-wnlx2"
Oct 30 22:09:19.811017 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:19.810988    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-344071f5-wnlx2"]
Oct 30 22:09:19.850646 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:19.850608    5678 patch_prober.go:29] interesting pod/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-jps22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Oct 30 22:09:19.850889 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:19.850865    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-jps22" podUID="d167328f-3684-41d7-9407-5ceec39dd03f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Oct 30 22:09:19.853712 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:19.853681    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:09:19.854153 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:19.853740    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5rvcj\""
Oct 30 22:09:19.864131 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:19.864105    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5191ae0-55f2-4180-b01e-226183a6a225-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-344071f5-wnlx2\" (UID: \"c5191ae0-55f2-4180-b01e-226183a6a225\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-344071f5-wnlx2"
Oct 30 22:09:19.864253 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:19.864135    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbb6n\" (UniqueName: \"kubernetes.io/projected/c5191ae0-55f2-4180-b01e-226183a6a225-kube-api-access-pbb6n\") pod \"maas-api-revoke-sub-344071f5-wnlx2\" (UID: \"c5191ae0-55f2-4180-b01e-226183a6a225\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-344071f5-wnlx2"
Oct 30 22:09:19.864253 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:19.864157    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5191ae0-55f2-4180-b01e-226183a6a225-maas-api-xks-ca\") pod \"maas-api-revoke-sub-344071f5-wnlx2\" (UID: \"c5191ae0-55f2-4180-b01e-226183a6a225\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-344071f5-wnlx2"
Oct 30 22:09:19.864841 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:19.864822    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5191ae0-55f2-4180-b01e-226183a6a225-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-344071f5-wnlx2\" (UID: \"c5191ae0-55f2-4180-b01e-226183a6a225\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-344071f5-wnlx2"
Oct 30 22:09:19.871017 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:19.870991    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5191ae0-55f2-4180-b01e-226183a6a225-maas-api-xks-ca\") pod \"maas-api-revoke-sub-344071f5-wnlx2\" (UID: \"c5191ae0-55f2-4180-b01e-226183a6a225\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-344071f5-wnlx2"
Oct 30 22:09:19.948849 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:19.948816    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbb6n\" (UniqueName: \"kubernetes.io/projected/c5191ae0-55f2-4180-b01e-226183a6a225-kube-api-access-pbb6n\") pod \"maas-api-revoke-sub-344071f5-wnlx2\" (UID: \"c5191ae0-55f2-4180-b01e-226183a6a225\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-344071f5-wnlx2"
Oct 30 22:09:20.069922 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:20.069840    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-344071f5-wnlx2"
Oct 30 22:09:20.472720 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:09:20.472679    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5191ae0_55f2_4180_b01e_226183a6a225.slice/crio-019d7c14b1b6355b45135d67904b5c713bfd09de690b64e247bd3c36d3ed3ac7 WatchSource:0}: Error finding container 019d7c14b1b6355b45135d67904b5c713bfd09de690b64e247bd3c36d3ed3ac7: Status 404 returned error can't find the container with id 019d7c14b1b6355b45135d67904b5c713bfd09de690b64e247bd3c36d3ed3ac7
Oct 30 22:09:20.473177 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:20.473151    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-344071f5-wnlx2"]
Oct 30 22:09:20.710501 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:20.710451    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-344071f5-wnlx2" event={"ID":"c5191ae0-55f2-4180-b01e-226183a6a225","Type":"ContainerStarted","Data":"019d7c14b1b6355b45135d67904b5c713bfd09de690b64e247bd3c36d3ed3ac7"}
Oct 30 22:09:20.850847 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:20.850819    5678 patch_prober.go:29] interesting pod/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-jps22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Oct 30 22:09:20.850921 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:20.850873    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-jps22" podUID="d167328f-3684-41d7-9407-5ceec39dd03f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Oct 30 22:09:21.720728 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:21.720696    5678 generic.go:356] "Generic (PLEG): container finished" podID="c5191ae0-55f2-4180-b01e-226183a6a225" containerID="60c895bad8fcf4df291affcfe3b4dcc47654b702a915ebaf1889d9d6886c8503" exitCode=0
Oct 30 22:09:21.721183 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:21.720777    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-344071f5-wnlx2" event={"ID":"c5191ae0-55f2-4180-b01e-226183a6a225","Type":"ContainerDied","Data":"60c895bad8fcf4df291affcfe3b4dcc47654b702a915ebaf1889d9d6886c8503"}
Oct 30 22:09:21.850920 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:21.850883    5678 patch_prober.go:29] interesting pod/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-jps22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Oct 30 22:09:21.851088 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:21.850946    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-jps22" podUID="d167328f-3684-41d7-9407-5ceec39dd03f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Oct 30 22:09:22.853159 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:22.850909    5678 patch_prober.go:29] interesting pod/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-jps22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Oct 30 22:09:22.853159 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:22.850984    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-jps22" podUID="d167328f-3684-41d7-9407-5ceec39dd03f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Oct 30 22:09:22.916985 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:22.916926    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-344071f5-wnlx2"
Oct 30 22:09:23.000145 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:23.000115    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5191ae0-55f2-4180-b01e-226183a6a225-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5191ae0-55f2-4180-b01e-226183a6a225-maas-api-xks-ca\") pod \"c5191ae0-55f2-4180-b01e-226183a6a225\" (UID: \"c5191ae0-55f2-4180-b01e-226183a6a225\") "
Oct 30 22:09:23.000333 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:23.000221    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5191ae0-55f2-4180-b01e-226183a6a225-kube-api-access-pbb6n\" (UniqueName: \"kubernetes.io/projected/c5191ae0-55f2-4180-b01e-226183a6a225-kube-api-access-pbb6n\") pod \"c5191ae0-55f2-4180-b01e-226183a6a225\" (UID: \"c5191ae0-55f2-4180-b01e-226183a6a225\") "
Oct 30 22:09:23.000333 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:23.000263    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5191ae0-55f2-4180-b01e-226183a6a225-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5191ae0-55f2-4180-b01e-226183a6a225-maas-api-openshift-ca\") pod \"c5191ae0-55f2-4180-b01e-226183a6a225\" (UID: \"c5191ae0-55f2-4180-b01e-226183a6a225\") "
Oct 30 22:09:23.000633 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:23.000605    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5191ae0-55f2-4180-b01e-226183a6a225-maas-api-openshift-ca" pod "c5191ae0-55f2-4180-b01e-226183a6a225" (UID: "c5191ae0-55f2-4180-b01e-226183a6a225"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:09:23.002239 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:23.002217    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5191ae0-55f2-4180-b01e-226183a6a225-maas-api-xks-ca" pod "c5191ae0-55f2-4180-b01e-226183a6a225" (UID: "c5191ae0-55f2-4180-b01e-226183a6a225"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:09:23.002323 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:23.002299    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5191ae0-55f2-4180-b01e-226183a6a225-kube-api-access-pbb6n" pod "c5191ae0-55f2-4180-b01e-226183a6a225" (UID: "c5191ae0-55f2-4180-b01e-226183a6a225"). InnerVolumeSpecName "kube-api-access-pbb6n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:09:23.101850 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:23.101817    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5191ae0-55f2-4180-b01e-226183a6a225-maas-api-openshift-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:09:23.101850 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:23.101847    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5191ae0-55f2-4180-b01e-226183a6a225-maas-api-xks-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:09:23.102036 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:23.101860    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbb6n\" (UniqueName: \"kubernetes.io/projected/c5191ae0-55f2-4180-b01e-226183a6a225-kube-api-access-pbb6n\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:09:23.759835 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:23.759801    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-344071f5-wnlx2"
Oct 30 22:09:23.759835 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:23.759827    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-344071f5-wnlx2" event={"ID":"c5191ae0-55f2-4180-b01e-226183a6a225","Type":"ContainerDied","Data":"019d7c14b1b6355b45135d67904b5c713bfd09de690b64e247bd3c36d3ed3ac7"}
Oct 30 22:09:23.760051 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:23.759877    5678 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="019d7c14b1b6355b45135d67904b5c713bfd09de690b64e247bd3c36d3ed3ac7"
Oct 30 22:09:23.850497 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:23.850467    5678 patch_prober.go:29] interesting pod/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-jps22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Oct 30 22:09:23.850650 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:23.850515    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-jps22" podUID="d167328f-3684-41d7-9407-5ceec39dd03f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Oct 30 22:09:24.853898 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:24.853869    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-jps22"
Oct 30 22:09:24.854332 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:24.854300    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-jps22"
Oct 30 22:09:24.855050 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:24.855027    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-jps22"
Oct 30 22:09:30.213665 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:30.213630    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-t6pm5"]
Oct 30 22:09:30.216079 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:30.214115    5678 cpu_manager.go:414] "containerMap: removing container" podUID="c5191ae0-55f2-4180-b01e-226183a6a225" containerName="revoke-keys"
Oct 30 22:09:30.216079 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:30.214125    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5191ae0-55f2-4180-b01e-226183a6a225" containerName="revoke-keys"
Oct 30 22:09:30.216079 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:30.214225    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5191ae0-55f2-4180-b01e-226183a6a225" containerName="revoke-keys"
Oct 30 22:09:30.217588 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:30.217572    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-t6pm5"
Oct 30 22:09:30.281128 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:30.281086    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-t6pm5"]
Oct 30 22:09:30.284050 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:30.284006    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d40decd-0eb0-4636-a8fe-06bbafd0484e-credential-socket\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-t6pm5\" (UID: \"5d40decd-0eb0-4636-a8fe-06bbafd0484e\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-t6pm5"
Oct 30 22:09:30.284232 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:30.284087    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvcpd\" (UniqueName: \"kubernetes.io/projected/5d40decd-0eb0-4636-a8fe-06bbafd0484e-kube-api-access-pvcpd\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-t6pm5\" (UID: \"5d40decd-0eb0-4636-a8fe-06bbafd0484e\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-t6pm5"
Oct 30 22:09:30.284232 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:30.284145    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d40decd-0eb0-4636-a8fe-06bbafd0484e-workload-certs\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-t6pm5\" (UID: \"5d40decd-0eb0-4636-a8fe-06bbafd0484e\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-t6pm5"
Oct 30 22:09:30.284232 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:30.284178    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d40decd-0eb0-4636-a8fe-06bbafd0484e-istio-token\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-t6pm5\" (UID: \"5d40decd-0eb0-4636-a8fe-06bbafd0484e\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-t6pm5"
Oct 30 22:09:30.284415 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:30.284235    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d40decd-0eb0-4636-a8fe-06bbafd0484e-istio-data\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-t6pm5\" (UID: \"5d40decd-0eb0-4636-a8fe-06bbafd0484e\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-t6pm5"
Oct 30 22:09:30.284415 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:30.284282    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d40decd-0eb0-4636-a8fe-06bbafd0484e-istiod-ca-cert\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-t6pm5\" (UID: \"5d40decd-0eb0-4636-a8fe-06bbafd0484e\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-t6pm5"
Oct 30 22:09:30.284415 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:30.284335    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d40decd-0eb0-4636-a8fe-06bbafd0484e-workload-socket\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-t6pm5\" (UID: \"5d40decd-0eb0-4636-a8fe-06bbafd0484e\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-t6pm5"
Oct 30 22:09:30.284415 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:30.284392    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d40decd-0eb0-4636-a8fe-06bbafd0484e-istio-podinfo\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-t6pm5\" (UID: \"5d40decd-0eb0-4636-a8fe-06bbafd0484e\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-t6pm5"
Oct 30 22:09:30.284614 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:30.284446    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d40decd-0eb0-4636-a8fe-06bbafd0484e-istio-envoy\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-t6pm5\" (UID: \"5d40decd-0eb0-4636-a8fe-06bbafd0484e\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-t6pm5"
Oct 30 22:09:30.385625 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:30.385590    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d40decd-0eb0-4636-a8fe-06bbafd0484e-istio-envoy\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-t6pm5\" (UID: \"5d40decd-0eb0-4636-a8fe-06bbafd0484e\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-t6pm5"
Oct 30 22:09:30.385797 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:30.385663    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d40decd-0eb0-4636-a8fe-06bbafd0484e-credential-socket\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-t6pm5\" (UID: \"5d40decd-0eb0-4636-a8fe-06bbafd0484e\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-t6pm5"
Oct 30 22:09:30.385797 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:30.385693    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvcpd\" (UniqueName: \"kubernetes.io/projected/5d40decd-0eb0-4636-a8fe-06bbafd0484e-kube-api-access-pvcpd\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-t6pm5\" (UID: \"5d40decd-0eb0-4636-a8fe-06bbafd0484e\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-t6pm5"
Oct 30 22:09:30.385797 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:30.385730    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d40decd-0eb0-4636-a8fe-06bbafd0484e-workload-certs\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-t6pm5\" (UID: \"5d40decd-0eb0-4636-a8fe-06bbafd0484e\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-t6pm5"
Oct 30 22:09:30.385797 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:30.385775    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d40decd-0eb0-4636-a8fe-06bbafd0484e-istio-token\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-t6pm5\" (UID: \"5d40decd-0eb0-4636-a8fe-06bbafd0484e\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-t6pm5"
Oct 30 22:09:30.385996 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:30.385812    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d40decd-0eb0-4636-a8fe-06bbafd0484e-istio-data\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-t6pm5\" (UID: \"5d40decd-0eb0-4636-a8fe-06bbafd0484e\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-t6pm5"
Oct 30 22:09:30.385996 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:30.385850    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d40decd-0eb0-4636-a8fe-06bbafd0484e-istiod-ca-cert\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-t6pm5\" (UID: \"5d40decd-0eb0-4636-a8fe-06bbafd0484e\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-t6pm5"
Oct 30 22:09:30.385996 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:30.385900    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d40decd-0eb0-4636-a8fe-06bbafd0484e-workload-socket\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-t6pm5\" (UID: \"5d40decd-0eb0-4636-a8fe-06bbafd0484e\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-t6pm5"
Oct 30 22:09:30.385996 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:30.385958    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d40decd-0eb0-4636-a8fe-06bbafd0484e-istio-podinfo\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-t6pm5\" (UID: \"5d40decd-0eb0-4636-a8fe-06bbafd0484e\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-t6pm5"
Oct 30 22:09:30.386209 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:30.386083    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d40decd-0eb0-4636-a8fe-06bbafd0484e-credential-socket\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-t6pm5\" (UID: \"5d40decd-0eb0-4636-a8fe-06bbafd0484e\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-t6pm5"
Oct 30 22:09:30.386209 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:30.386183    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d40decd-0eb0-4636-a8fe-06bbafd0484e-workload-certs\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-t6pm5\" (UID: \"5d40decd-0eb0-4636-a8fe-06bbafd0484e\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-t6pm5"
Oct 30 22:09:30.386318 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:30.386288    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d40decd-0eb0-4636-a8fe-06bbafd0484e-workload-socket\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-t6pm5\" (UID: \"5d40decd-0eb0-4636-a8fe-06bbafd0484e\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-t6pm5"
Oct 30 22:09:30.386377 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:30.386358    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d40decd-0eb0-4636-a8fe-06bbafd0484e-istio-data\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-t6pm5\" (UID: \"5d40decd-0eb0-4636-a8fe-06bbafd0484e\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-t6pm5"
Oct 30 22:09:30.386685 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:30.386657    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d40decd-0eb0-4636-a8fe-06bbafd0484e-istiod-ca-cert\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-t6pm5\" (UID: \"5d40decd-0eb0-4636-a8fe-06bbafd0484e\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-t6pm5"
Oct 30 22:09:30.387909 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:30.387889    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d40decd-0eb0-4636-a8fe-06bbafd0484e-istio-envoy\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-t6pm5\" (UID: \"5d40decd-0eb0-4636-a8fe-06bbafd0484e\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-t6pm5"
Oct 30 22:09:30.388371 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:30.388355    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d40decd-0eb0-4636-a8fe-06bbafd0484e-istio-podinfo\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-t6pm5\" (UID: \"5d40decd-0eb0-4636-a8fe-06bbafd0484e\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-t6pm5"
Oct 30 22:09:30.450082 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:30.450054    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d40decd-0eb0-4636-a8fe-06bbafd0484e-istio-token\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-t6pm5\" (UID: \"5d40decd-0eb0-4636-a8fe-06bbafd0484e\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-t6pm5"
Oct 30 22:09:30.450202 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:30.450184    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvcpd\" (UniqueName: \"kubernetes.io/projected/5d40decd-0eb0-4636-a8fe-06bbafd0484e-kube-api-access-pvcpd\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-t6pm5\" (UID: \"5d40decd-0eb0-4636-a8fe-06bbafd0484e\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-t6pm5"
Oct 30 22:09:30.531985 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:30.531913    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-t6pm5"
Oct 30 22:09:30.876905 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:30.876875    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-fpsll"]
Oct 30 22:09:30.882069 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:30.882050    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-fpsll"
Oct 30 22:09:30.969381 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:09:30.969343    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d40decd_0eb0_4636_a8fe_06bbafd0484e.slice/crio-2e005984eacd3975b692661a5d974e32a5845c06a30417816e15c39988dffd53 WatchSource:0}: Error finding container 2e005984eacd3975b692661a5d974e32a5845c06a30417816e15c39988dffd53: Status 404 returned error can't find the container with id 2e005984eacd3975b692661a5d974e32a5845c06a30417816e15c39988dffd53
Oct 30 22:09:30.972009 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:30.971979    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:09:30.972113 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:30.972053    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:09:30.972113 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:30.972081    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:09:30.992034 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:30.991996    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-t6pm5"]
Oct 30 22:09:30.994098 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:30.994068    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52992d7b-821d-4b13-945c-d0b9b59ddc58-istio-token\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-fpsll\" (UID: \"52992d7b-821d-4b13-945c-d0b9b59ddc58\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-fpsll"
Oct 30 22:09:30.994227 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:30.994206    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52992d7b-821d-4b13-945c-d0b9b59ddc58-workload-socket\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-fpsll\" (UID: \"52992d7b-821d-4b13-945c-d0b9b59ddc58\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-fpsll"
Oct 30 22:09:30.994288 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:30.994244    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52992d7b-821d-4b13-945c-d0b9b59ddc58-istiod-ca-cert\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-fpsll\" (UID: \"52992d7b-821d-4b13-945c-d0b9b59ddc58\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-fpsll"
Oct 30 22:09:30.994363 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:30.994338    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52992d7b-821d-4b13-945c-d0b9b59ddc58-credential-socket\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-fpsll\" (UID: \"52992d7b-821d-4b13-945c-d0b9b59ddc58\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-fpsll"
Oct 30 22:09:30.994454 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:30.994391    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52992d7b-821d-4b13-945c-d0b9b59ddc58-istio-podinfo\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-fpsll\" (UID: \"52992d7b-821d-4b13-945c-d0b9b59ddc58\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-fpsll"
Oct 30 22:09:30.994454 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:30.994438    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rf2s2\" (UniqueName: \"kubernetes.io/projected/52992d7b-821d-4b13-945c-d0b9b59ddc58-kube-api-access-rf2s2\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-fpsll\" (UID: \"52992d7b-821d-4b13-945c-d0b9b59ddc58\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-fpsll"
Oct 30 22:09:30.994560 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:30.994474    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52992d7b-821d-4b13-945c-d0b9b59ddc58-istio-data\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-fpsll\" (UID: \"52992d7b-821d-4b13-945c-d0b9b59ddc58\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-fpsll"
Oct 30 22:09:30.994560 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:30.994510    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52992d7b-821d-4b13-945c-d0b9b59ddc58-istio-envoy\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-fpsll\" (UID: \"52992d7b-821d-4b13-945c-d0b9b59ddc58\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-fpsll"
Oct 30 22:09:30.994679 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:30.994664    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52992d7b-821d-4b13-945c-d0b9b59ddc58-workload-certs\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-fpsll\" (UID: \"52992d7b-821d-4b13-945c-d0b9b59ddc58\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-fpsll"
Oct 30 22:09:30.996018 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:30.995992    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-fpsll"]
Oct 30 22:09:31.095592 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:31.095560    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52992d7b-821d-4b13-945c-d0b9b59ddc58-workload-socket\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-fpsll\" (UID: \"52992d7b-821d-4b13-945c-d0b9b59ddc58\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-fpsll"
Oct 30 22:09:31.095592 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:31.095601    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52992d7b-821d-4b13-945c-d0b9b59ddc58-istiod-ca-cert\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-fpsll\" (UID: \"52992d7b-821d-4b13-945c-d0b9b59ddc58\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-fpsll"
Oct 30 22:09:31.095855 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:31.095639    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52992d7b-821d-4b13-945c-d0b9b59ddc58-credential-socket\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-fpsll\" (UID: \"52992d7b-821d-4b13-945c-d0b9b59ddc58\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-fpsll"
Oct 30 22:09:31.095855 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:31.095676    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52992d7b-821d-4b13-945c-d0b9b59ddc58-istio-podinfo\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-fpsll\" (UID: \"52992d7b-821d-4b13-945c-d0b9b59ddc58\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-fpsll"
Oct 30 22:09:31.095855 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:31.095706    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rf2s2\" (UniqueName: \"kubernetes.io/projected/52992d7b-821d-4b13-945c-d0b9b59ddc58-kube-api-access-rf2s2\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-fpsll\" (UID: \"52992d7b-821d-4b13-945c-d0b9b59ddc58\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-fpsll"
Oct 30 22:09:31.095855 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:31.095744    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52992d7b-821d-4b13-945c-d0b9b59ddc58-istio-data\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-fpsll\" (UID: \"52992d7b-821d-4b13-945c-d0b9b59ddc58\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-fpsll"
Oct 30 22:09:31.095855 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:31.095775    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52992d7b-821d-4b13-945c-d0b9b59ddc58-istio-envoy\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-fpsll\" (UID: \"52992d7b-821d-4b13-945c-d0b9b59ddc58\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-fpsll"
Oct 30 22:09:31.096139 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:31.096022    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52992d7b-821d-4b13-945c-d0b9b59ddc58-workload-certs\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-fpsll\" (UID: \"52992d7b-821d-4b13-945c-d0b9b59ddc58\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-fpsll"
Oct 30 22:09:31.096139 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:31.096058    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52992d7b-821d-4b13-945c-d0b9b59ddc58-istio-token\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-fpsll\" (UID: \"52992d7b-821d-4b13-945c-d0b9b59ddc58\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-fpsll"
Oct 30 22:09:31.096139 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:31.096083    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52992d7b-821d-4b13-945c-d0b9b59ddc58-workload-socket\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-fpsll\" (UID: \"52992d7b-821d-4b13-945c-d0b9b59ddc58\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-fpsll"
Oct 30 22:09:31.096139 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:31.096118    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52992d7b-821d-4b13-945c-d0b9b59ddc58-credential-socket\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-fpsll\" (UID: \"52992d7b-821d-4b13-945c-d0b9b59ddc58\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-fpsll"
Oct 30 22:09:31.096481 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:31.096456    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52992d7b-821d-4b13-945c-d0b9b59ddc58-istio-data\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-fpsll\" (UID: \"52992d7b-821d-4b13-945c-d0b9b59ddc58\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-fpsll"
Oct 30 22:09:31.096547 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:31.096485    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52992d7b-821d-4b13-945c-d0b9b59ddc58-istiod-ca-cert\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-fpsll\" (UID: \"52992d7b-821d-4b13-945c-d0b9b59ddc58\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-fpsll"
Oct 30 22:09:31.096805 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:31.096785    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52992d7b-821d-4b13-945c-d0b9b59ddc58-workload-certs\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-fpsll\" (UID: \"52992d7b-821d-4b13-945c-d0b9b59ddc58\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-fpsll"
Oct 30 22:09:31.098491 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:31.098473    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52992d7b-821d-4b13-945c-d0b9b59ddc58-istio-envoy\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-fpsll\" (UID: \"52992d7b-821d-4b13-945c-d0b9b59ddc58\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-fpsll"
Oct 30 22:09:31.098817 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:31.098791    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52992d7b-821d-4b13-945c-d0b9b59ddc58-istio-podinfo\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-fpsll\" (UID: \"52992d7b-821d-4b13-945c-d0b9b59ddc58\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-fpsll"
Oct 30 22:09:31.155296 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:31.155231    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52992d7b-821d-4b13-945c-d0b9b59ddc58-istio-token\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-fpsll\" (UID: \"52992d7b-821d-4b13-945c-d0b9b59ddc58\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-fpsll"
Oct 30 22:09:31.155597 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:31.155576    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rf2s2\" (UniqueName: \"kubernetes.io/projected/52992d7b-821d-4b13-945c-d0b9b59ddc58-kube-api-access-rf2s2\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-fpsll\" (UID: \"52992d7b-821d-4b13-945c-d0b9b59ddc58\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-fpsll"
Oct 30 22:09:31.197173 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:31.197135    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-fpsll"
Oct 30 22:09:31.677853 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:09:31.677817    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52992d7b_821d_4b13_945c_d0b9b59ddc58.slice/crio-179fb133649751bb685ace2e0c7ede726009431642a8d86372197ab8c826cdf2 WatchSource:0}: Error finding container 179fb133649751bb685ace2e0c7ede726009431642a8d86372197ab8c826cdf2: Status 404 returned error can't find the container with id 179fb133649751bb685ace2e0c7ede726009431642a8d86372197ab8c826cdf2
Oct 30 22:09:31.678293 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:31.678269    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-fpsll"]
Oct 30 22:09:31.680500 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:31.680469    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:09:31.680581 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:31.680533    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:09:31.680581 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:31.680564    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:09:31.813660 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:31.813611    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-t6pm5" event={"ID":"5d40decd-0eb0-4636-a8fe-06bbafd0484e","Type":"ContainerStarted","Data":"51e46e5c32fdd4104f6d5d325844c9f3a4051be47038c10876995bee67280709"}
Oct 30 22:09:31.813971 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:31.813946    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-t6pm5" event={"ID":"5d40decd-0eb0-4636-a8fe-06bbafd0484e","Type":"ContainerStarted","Data":"2e005984eacd3975b692661a5d974e32a5845c06a30417816e15c39988dffd53"}
Oct 30 22:09:31.815877 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:31.815847    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-fpsll" event={"ID":"52992d7b-821d-4b13-945c-d0b9b59ddc58","Type":"ContainerStarted","Data":"baa9ce10b5014f02d6861aa75dfc8069cbb7fa6cf3d720e9bdd79818ffa1c287"}
Oct 30 22:09:31.815976 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:31.815900    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-fpsll" event={"ID":"52992d7b-821d-4b13-945c-d0b9b59ddc58","Type":"ContainerStarted","Data":"179fb133649751bb685ace2e0c7ede726009431642a8d86372197ab8c826cdf2"}
Oct 30 22:09:31.951796 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:31.951673    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-t6pm5" podStartSLOduration=1.951652883 podStartE2EDuration="1.951652883s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:09:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:09:31.946024571 +0000 UTC m=+1149.550332125" watchObservedRunningTime="2026-10-30 22:09:31.951652883 +0000 UTC m=+1149.555960440"
Oct 30 22:09:32.063175 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:32.063118    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-fpsll" podStartSLOduration=2.063101465 podStartE2EDuration="2.063101465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:09:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:09:32.06246183 +0000 UTC m=+1149.666769403" watchObservedRunningTime="2026-10-30 22:09:32.063101465 +0000 UTC m=+1149.667409069"
Oct 30 22:09:32.198181 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:32.198148    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-fpsll"
Oct 30 22:09:32.200210 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:32.200180    5678 patch_prober.go:29] interesting pod/e2e-worker-w2-bc714d-openshift-default-67558b95cf-fpsll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Oct 30 22:09:32.200366 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:32.200235    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-fpsll" podUID="52992d7b-821d-4b13-945c-d0b9b59ddc58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Oct 30 22:09:32.533424 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:32.532692    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-t6pm5"
Oct 30 22:09:32.563424 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:32.546562    5678 patch_prober.go:29] interesting pod/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-t6pm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Oct 30 22:09:32.563424 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:32.546639    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-t6pm5" podUID="5d40decd-0eb0-4636-a8fe-06bbafd0484e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Oct 30 22:09:33.198020 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:33.197977    5678 patch_prober.go:29] interesting pod/e2e-worker-w2-bc714d-openshift-default-67558b95cf-fpsll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Oct 30 22:09:33.198443 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:33.198034    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-fpsll" podUID="52992d7b-821d-4b13-945c-d0b9b59ddc58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Oct 30 22:09:33.541495 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:33.533777    5678 patch_prober.go:29] interesting pod/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-t6pm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Oct 30 22:09:33.541495 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:33.533839    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-t6pm5" podUID="5d40decd-0eb0-4636-a8fe-06bbafd0484e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Oct 30 22:09:34.198690 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:34.198650    5678 patch_prober.go:29] interesting pod/e2e-worker-w2-bc714d-openshift-default-67558b95cf-fpsll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Oct 30 22:09:34.199104 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:34.198705    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-fpsll" podUID="52992d7b-821d-4b13-945c-d0b9b59ddc58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Oct 30 22:09:34.546478 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:34.545806    5678 patch_prober.go:29] interesting pod/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-t6pm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Oct 30 22:09:34.546478 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:34.545875    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-t6pm5" podUID="5d40decd-0eb0-4636-a8fe-06bbafd0484e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Oct 30 22:09:35.198581 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:35.198544    5678 patch_prober.go:29] interesting pod/e2e-worker-w2-bc714d-openshift-default-67558b95cf-fpsll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Oct 30 22:09:35.198760 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:35.198603    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-fpsll" podUID="52992d7b-821d-4b13-945c-d0b9b59ddc58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Oct 30 22:09:35.538291 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:35.538201    5678 patch_prober.go:29] interesting pod/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-t6pm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Oct 30 22:09:35.538585 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:35.538555    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-t6pm5" podUID="5d40decd-0eb0-4636-a8fe-06bbafd0484e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Oct 30 22:09:35.863648 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:35.863553    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0794c9d6-jjsx9"]
Oct 30 22:09:35.869253 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:35.869229    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0794c9d6-jjsx9"
Oct 30 22:09:35.906085 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:35.906050    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0794c9d6-jjsx9"]
Oct 30 22:09:35.950425 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:35.946448    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5rvcj\""
Oct 30 22:09:35.959486 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:35.946512    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:09:35.997661 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:35.997629    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/add39a5f-865c-4416-9877-d2fcbe815b47-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0794c9d6-jjsx9\" (UID: \"add39a5f-865c-4416-9877-d2fcbe815b47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0794c9d6-jjsx9"
Oct 30 22:09:35.997796 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:35.997760    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpkfw\" (UniqueName: \"kubernetes.io/projected/add39a5f-865c-4416-9877-d2fcbe815b47-kube-api-access-tpkfw\") pod \"maas-api-revoke-sub-0794c9d6-jjsx9\" (UID: \"add39a5f-865c-4416-9877-d2fcbe815b47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0794c9d6-jjsx9"
Oct 30 22:09:35.997796 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:35.997785    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/add39a5f-865c-4416-9877-d2fcbe815b47-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0794c9d6-jjsx9\" (UID: \"add39a5f-865c-4416-9877-d2fcbe815b47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0794c9d6-jjsx9"
Oct 30 22:09:36.099508 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:36.099471    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpkfw\" (UniqueName: \"kubernetes.io/projected/add39a5f-865c-4416-9877-d2fcbe815b47-kube-api-access-tpkfw\") pod \"maas-api-revoke-sub-0794c9d6-jjsx9\" (UID: \"add39a5f-865c-4416-9877-d2fcbe815b47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0794c9d6-jjsx9"
Oct 30 22:09:36.099659 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:36.099522    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/add39a5f-865c-4416-9877-d2fcbe815b47-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0794c9d6-jjsx9\" (UID: \"add39a5f-865c-4416-9877-d2fcbe815b47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0794c9d6-jjsx9"
Oct 30 22:09:36.099659 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:36.099592    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/add39a5f-865c-4416-9877-d2fcbe815b47-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0794c9d6-jjsx9\" (UID: \"add39a5f-865c-4416-9877-d2fcbe815b47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0794c9d6-jjsx9"
Oct 30 22:09:36.100381 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:36.100360    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/add39a5f-865c-4416-9877-d2fcbe815b47-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0794c9d6-jjsx9\" (UID: \"add39a5f-865c-4416-9877-d2fcbe815b47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0794c9d6-jjsx9"
Oct 30 22:09:36.101971 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:36.101951    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/add39a5f-865c-4416-9877-d2fcbe815b47-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0794c9d6-jjsx9\" (UID: \"add39a5f-865c-4416-9877-d2fcbe815b47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0794c9d6-jjsx9"
Oct 30 22:09:36.175097 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:36.175057    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpkfw\" (UniqueName: \"kubernetes.io/projected/add39a5f-865c-4416-9877-d2fcbe815b47-kube-api-access-tpkfw\") pod \"maas-api-revoke-sub-0794c9d6-jjsx9\" (UID: \"add39a5f-865c-4416-9877-d2fcbe815b47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0794c9d6-jjsx9"
Oct 30 22:09:36.188464 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:36.188433    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0794c9d6-jjsx9"
Oct 30 22:09:36.197680 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:36.197655    5678 patch_prober.go:29] interesting pod/e2e-worker-w2-bc714d-openshift-default-67558b95cf-fpsll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Oct 30 22:09:36.197806 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:36.197707    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-fpsll" podUID="52992d7b-821d-4b13-945c-d0b9b59ddc58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Oct 30 22:09:36.543479 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:36.543228    5678 patch_prober.go:29] interesting pod/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-t6pm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Oct 30 22:09:36.543479 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:36.543297    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-t6pm5" podUID="5d40decd-0eb0-4636-a8fe-06bbafd0484e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Oct 30 22:09:36.656441 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:36.645940    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0794c9d6-jjsx9"]
Oct 30 22:09:36.673962 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:09:36.673911    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadd39a5f_865c_4416_9877_d2fcbe815b47.slice/crio-65df9162cc259b446bb2cf6b70161af2033665a78a4a0e92f3272f201f3c21af WatchSource:0}: Error finding container 65df9162cc259b446bb2cf6b70161af2033665a78a4a0e92f3272f201f3c21af: Status 404 returned error can't find the container with id 65df9162cc259b446bb2cf6b70161af2033665a78a4a0e92f3272f201f3c21af
Oct 30 22:09:36.904686 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:36.904646    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0794c9d6-jjsx9" event={"ID":"add39a5f-865c-4416-9877-d2fcbe815b47","Type":"ContainerStarted","Data":"65df9162cc259b446bb2cf6b70161af2033665a78a4a0e92f3272f201f3c21af"}
Oct 30 22:09:37.199484 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:37.198922    5678 patch_prober.go:29] interesting pod/e2e-worker-w2-bc714d-openshift-default-67558b95cf-fpsll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Oct 30 22:09:37.199484 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:37.198986    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-fpsll" podUID="52992d7b-821d-4b13-945c-d0b9b59ddc58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Oct 30 22:09:37.536469 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:37.535743    5678 patch_prober.go:29] interesting pod/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-t6pm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Oct 30 22:09:37.536469 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:37.535803    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-t6pm5" podUID="5d40decd-0eb0-4636-a8fe-06bbafd0484e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Oct 30 22:09:37.988173 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:37.988133    5678 generic.go:356] "Generic (PLEG): container finished" podID="add39a5f-865c-4416-9877-d2fcbe815b47" containerID="713bf6c9cb2819553273cfe492307b86dd6fcb41e27ac925177c78a7cdaf340e" exitCode=0
Oct 30 22:09:37.988906 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:37.988322    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0794c9d6-jjsx9" event={"ID":"add39a5f-865c-4416-9877-d2fcbe815b47","Type":"ContainerDied","Data":"713bf6c9cb2819553273cfe492307b86dd6fcb41e27ac925177c78a7cdaf340e"}
Oct 30 22:09:38.202393 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:38.202359    5678 patch_prober.go:29] interesting pod/e2e-worker-w2-bc714d-openshift-default-67558b95cf-fpsll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Oct 30 22:09:38.202720 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:38.202673    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-fpsll" podUID="52992d7b-821d-4b13-945c-d0b9b59ddc58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Oct 30 22:09:38.588190 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:38.588098    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-t6pm5"
Oct 30 22:09:38.588625 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:38.588600    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-t6pm5"
Oct 30 22:09:38.589614 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:38.589588    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-t6pm5"
Oct 30 22:09:39.176939 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:39.176913    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0794c9d6-jjsx9"
Oct 30 22:09:39.218433 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:39.218386    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/add39a5f-865c-4416-9877-d2fcbe815b47-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/add39a5f-865c-4416-9877-d2fcbe815b47-maas-api-openshift-ca\") pod \"add39a5f-865c-4416-9877-d2fcbe815b47\" (UID: \"add39a5f-865c-4416-9877-d2fcbe815b47\") "
Oct 30 22:09:39.218570 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:39.218488    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/add39a5f-865c-4416-9877-d2fcbe815b47-kube-api-access-tpkfw\" (UniqueName: \"kubernetes.io/projected/add39a5f-865c-4416-9877-d2fcbe815b47-kube-api-access-tpkfw\") pod \"add39a5f-865c-4416-9877-d2fcbe815b47\" (UID: \"add39a5f-865c-4416-9877-d2fcbe815b47\") "
Oct 30 22:09:39.218672 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:39.218652    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/add39a5f-865c-4416-9877-d2fcbe815b47-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/add39a5f-865c-4416-9877-d2fcbe815b47-maas-api-xks-ca\") pod \"add39a5f-865c-4416-9877-d2fcbe815b47\" (UID: \"add39a5f-865c-4416-9877-d2fcbe815b47\") "
Oct 30 22:09:39.218741 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:39.218706    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/add39a5f-865c-4416-9877-d2fcbe815b47-maas-api-openshift-ca" pod "add39a5f-865c-4416-9877-d2fcbe815b47" (UID: "add39a5f-865c-4416-9877-d2fcbe815b47"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:09:39.219181 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:39.219156    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/add39a5f-865c-4416-9877-d2fcbe815b47-maas-api-openshift-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:09:39.220552 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:39.220525    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/add39a5f-865c-4416-9877-d2fcbe815b47-kube-api-access-tpkfw" pod "add39a5f-865c-4416-9877-d2fcbe815b47" (UID: "add39a5f-865c-4416-9877-d2fcbe815b47"). InnerVolumeSpecName "kube-api-access-tpkfw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:09:39.220552 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:39.220541    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/add39a5f-865c-4416-9877-d2fcbe815b47-maas-api-xks-ca" pod "add39a5f-865c-4416-9877-d2fcbe815b47" (UID: "add39a5f-865c-4416-9877-d2fcbe815b47"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:09:39.321427 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:39.321323    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/add39a5f-865c-4416-9877-d2fcbe815b47-maas-api-xks-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:09:39.321427 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:39.321364    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpkfw\" (UniqueName: \"kubernetes.io/projected/add39a5f-865c-4416-9877-d2fcbe815b47-kube-api-access-tpkfw\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:09:39.749882 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:39.749846    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-fpsll"
Oct 30 22:09:39.750264 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:39.750244    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-fpsll"
Oct 30 22:09:39.751393 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:39.751370    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-fpsll"
Oct 30 22:09:40.007852 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:40.007776    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0794c9d6-jjsx9"
Oct 30 22:09:40.008036 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:40.007791    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0794c9d6-jjsx9" event={"ID":"add39a5f-865c-4416-9877-d2fcbe815b47","Type":"ContainerDied","Data":"65df9162cc259b446bb2cf6b70161af2033665a78a4a0e92f3272f201f3c21af"}
Oct 30 22:09:40.008171 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:09:40.008156    5678 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="65df9162cc259b446bb2cf6b70161af2033665a78a4a0e92f3272f201f3c21af"
Oct 30 22:09:40.570577 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:09:40.570540    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 22:10:01.283305 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:01.283224    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-7bb1c2/distinct-w3-7bb1c2-kserve-8d695596c-8zgtw"]
Oct 30 22:10:01.283846 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:01.283827    5678 cpu_manager.go:414] "containerMap: removing container" podUID="add39a5f-865c-4416-9877-d2fcbe815b47" containerName="revoke-keys"
Oct 30 22:10:01.283894 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:01.283849    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="add39a5f-865c-4416-9877-d2fcbe815b47" containerName="revoke-keys"
Oct 30 22:10:01.284079 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:01.284055    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="add39a5f-865c-4416-9877-d2fcbe815b47" containerName="revoke-keys"
Oct 30 22:10:01.289069 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:01.289052    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-7bb1c2/distinct-w3-7bb1c2-kserve-8d695596c-8zgtw"
Oct 30 22:10:01.386765 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:01.386733    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-7bb1c2\"/\"distinct-w3-7bb1c2-kserve-self-signed-certs\""
Oct 30 22:10:01.390374 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:01.390352    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-7bb1c2/distinct-w3-7bb1c2-kserve-8d695596c-8zgtw"]
Oct 30 22:10:01.395983 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:01.395962    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36f0a6a2-54be-49bf-9202-502960cbb134-home\") pod \"distinct-w3-7bb1c2-kserve-8d695596c-8zgtw\" (UID: \"36f0a6a2-54be-49bf-9202-502960cbb134\") " pod="e2e-models-e2e-worker-w3-7bb1c2/distinct-w3-7bb1c2-kserve-8d695596c-8zgtw"
Oct 30 22:10:01.396095 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:01.395993    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36f0a6a2-54be-49bf-9202-502960cbb134-model-cache\") pod \"distinct-w3-7bb1c2-kserve-8d695596c-8zgtw\" (UID: \"36f0a6a2-54be-49bf-9202-502960cbb134\") " pod="e2e-models-e2e-worker-w3-7bb1c2/distinct-w3-7bb1c2-kserve-8d695596c-8zgtw"
Oct 30 22:10:01.396095 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:01.396028    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36f0a6a2-54be-49bf-9202-502960cbb134-dshm\") pod \"distinct-w3-7bb1c2-kserve-8d695596c-8zgtw\" (UID: \"36f0a6a2-54be-49bf-9202-502960cbb134\") " pod="e2e-models-e2e-worker-w3-7bb1c2/distinct-w3-7bb1c2-kserve-8d695596c-8zgtw"
Oct 30 22:10:01.396192 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:01.396094    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36f0a6a2-54be-49bf-9202-502960cbb134-tmp-dir\") pod \"distinct-w3-7bb1c2-kserve-8d695596c-8zgtw\" (UID: \"36f0a6a2-54be-49bf-9202-502960cbb134\") " pod="e2e-models-e2e-worker-w3-7bb1c2/distinct-w3-7bb1c2-kserve-8d695596c-8zgtw"
Oct 30 22:10:01.396234 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:01.396215    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcfq5\" (UniqueName: \"kubernetes.io/projected/36f0a6a2-54be-49bf-9202-502960cbb134-kube-api-access-wcfq5\") pod \"distinct-w3-7bb1c2-kserve-8d695596c-8zgtw\" (UID: \"36f0a6a2-54be-49bf-9202-502960cbb134\") " pod="e2e-models-e2e-worker-w3-7bb1c2/distinct-w3-7bb1c2-kserve-8d695596c-8zgtw"
Oct 30 22:10:01.396272 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:01.396241    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36f0a6a2-54be-49bf-9202-502960cbb134-tls-certs\") pod \"distinct-w3-7bb1c2-kserve-8d695596c-8zgtw\" (UID: \"36f0a6a2-54be-49bf-9202-502960cbb134\") " pod="e2e-models-e2e-worker-w3-7bb1c2/distinct-w3-7bb1c2-kserve-8d695596c-8zgtw"
Oct 30 22:10:01.498072 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:01.498002    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36f0a6a2-54be-49bf-9202-502960cbb134-home\") pod \"distinct-w3-7bb1c2-kserve-8d695596c-8zgtw\" (UID: \"36f0a6a2-54be-49bf-9202-502960cbb134\") " pod="e2e-models-e2e-worker-w3-7bb1c2/distinct-w3-7bb1c2-kserve-8d695596c-8zgtw"
Oct 30 22:10:01.498072 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:01.498059    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36f0a6a2-54be-49bf-9202-502960cbb134-model-cache\") pod \"distinct-w3-7bb1c2-kserve-8d695596c-8zgtw\" (UID: \"36f0a6a2-54be-49bf-9202-502960cbb134\") " pod="e2e-models-e2e-worker-w3-7bb1c2/distinct-w3-7bb1c2-kserve-8d695596c-8zgtw"
Oct 30 22:10:01.498511 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:01.498095    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36f0a6a2-54be-49bf-9202-502960cbb134-dshm\") pod \"distinct-w3-7bb1c2-kserve-8d695596c-8zgtw\" (UID: \"36f0a6a2-54be-49bf-9202-502960cbb134\") " pod="e2e-models-e2e-worker-w3-7bb1c2/distinct-w3-7bb1c2-kserve-8d695596c-8zgtw"
Oct 30 22:10:01.498511 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:01.498116    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36f0a6a2-54be-49bf-9202-502960cbb134-tmp-dir\") pod \"distinct-w3-7bb1c2-kserve-8d695596c-8zgtw\" (UID: \"36f0a6a2-54be-49bf-9202-502960cbb134\") " pod="e2e-models-e2e-worker-w3-7bb1c2/distinct-w3-7bb1c2-kserve-8d695596c-8zgtw"
Oct 30 22:10:01.498511 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:01.498160    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcfq5\" (UniqueName: \"kubernetes.io/projected/36f0a6a2-54be-49bf-9202-502960cbb134-kube-api-access-wcfq5\") pod \"distinct-w3-7bb1c2-kserve-8d695596c-8zgtw\" (UID: \"36f0a6a2-54be-49bf-9202-502960cbb134\") " pod="e2e-models-e2e-worker-w3-7bb1c2/distinct-w3-7bb1c2-kserve-8d695596c-8zgtw"
Oct 30 22:10:01.498511 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:01.498186    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36f0a6a2-54be-49bf-9202-502960cbb134-tls-certs\") pod \"distinct-w3-7bb1c2-kserve-8d695596c-8zgtw\" (UID: \"36f0a6a2-54be-49bf-9202-502960cbb134\") " pod="e2e-models-e2e-worker-w3-7bb1c2/distinct-w3-7bb1c2-kserve-8d695596c-8zgtw"
Oct 30 22:10:01.499334 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:01.499301    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36f0a6a2-54be-49bf-9202-502960cbb134-tmp-dir\") pod \"distinct-w3-7bb1c2-kserve-8d695596c-8zgtw\" (UID: \"36f0a6a2-54be-49bf-9202-502960cbb134\") " pod="e2e-models-e2e-worker-w3-7bb1c2/distinct-w3-7bb1c2-kserve-8d695596c-8zgtw"
Oct 30 22:10:01.499814 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:01.499788    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36f0a6a2-54be-49bf-9202-502960cbb134-model-cache\") pod \"distinct-w3-7bb1c2-kserve-8d695596c-8zgtw\" (UID: \"36f0a6a2-54be-49bf-9202-502960cbb134\") " pod="e2e-models-e2e-worker-w3-7bb1c2/distinct-w3-7bb1c2-kserve-8d695596c-8zgtw"
Oct 30 22:10:01.500134 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:01.500115    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36f0a6a2-54be-49bf-9202-502960cbb134-home\") pod \"distinct-w3-7bb1c2-kserve-8d695596c-8zgtw\" (UID: \"36f0a6a2-54be-49bf-9202-502960cbb134\") " pod="e2e-models-e2e-worker-w3-7bb1c2/distinct-w3-7bb1c2-kserve-8d695596c-8zgtw"
Oct 30 22:10:01.501884 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:01.501857    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36f0a6a2-54be-49bf-9202-502960cbb134-dshm\") pod \"distinct-w3-7bb1c2-kserve-8d695596c-8zgtw\" (UID: \"36f0a6a2-54be-49bf-9202-502960cbb134\") " pod="e2e-models-e2e-worker-w3-7bb1c2/distinct-w3-7bb1c2-kserve-8d695596c-8zgtw"
Oct 30 22:10:01.501990 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:01.501942    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36f0a6a2-54be-49bf-9202-502960cbb134-tls-certs\") pod \"distinct-w3-7bb1c2-kserve-8d695596c-8zgtw\" (UID: \"36f0a6a2-54be-49bf-9202-502960cbb134\") " pod="e2e-models-e2e-worker-w3-7bb1c2/distinct-w3-7bb1c2-kserve-8d695596c-8zgtw"
Oct 30 22:10:01.563145 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:01.563073    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcfq5\" (UniqueName: \"kubernetes.io/projected/36f0a6a2-54be-49bf-9202-502960cbb134-kube-api-access-wcfq5\") pod \"distinct-w3-7bb1c2-kserve-8d695596c-8zgtw\" (UID: \"36f0a6a2-54be-49bf-9202-502960cbb134\") " pod="e2e-models-e2e-worker-w3-7bb1c2/distinct-w3-7bb1c2-kserve-8d695596c-8zgtw"
Oct 30 22:10:01.601182 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:01.601157    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-7bb1c2/distinct-w3-7bb1c2-kserve-8d695596c-8zgtw"
Oct 30 22:10:02.010626 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:02.010592    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-7bb1c2/distinct-w3-7bb1c2-kserve-8d695596c-8zgtw"]
Oct 30 22:10:02.010912 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:10:02.010883    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod36f0a6a2_54be_49bf_9202_502960cbb134.slice/crio-9fbfaf334ea0832a94b4c4d520b277f8eb1d1487462328dbfadcab2747fe9454 WatchSource:0}: Error finding container 9fbfaf334ea0832a94b4c4d520b277f8eb1d1487462328dbfadcab2747fe9454: Status 404 returned error can't find the container with id 9fbfaf334ea0832a94b4c4d520b277f8eb1d1487462328dbfadcab2747fe9454
Oct 30 22:10:02.163638 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:02.163604    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-7bb1c2/distinct-w3-7bb1c2-kserve-8d695596c-8zgtw" event={"ID":"36f0a6a2-54be-49bf-9202-502960cbb134","Type":"ContainerStarted","Data":"bd70d8c9ea9e84fb68044057299853590dd9d43ce9575a0bab5a3ba895b8f503"}
Oct 30 22:10:02.163811 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:02.163647    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-7bb1c2/distinct-w3-7bb1c2-kserve-8d695596c-8zgtw" event={"ID":"36f0a6a2-54be-49bf-9202-502960cbb134","Type":"ContainerStarted","Data":"9fbfaf334ea0832a94b4c4d520b277f8eb1d1487462328dbfadcab2747fe9454"}
Oct 30 22:10:02.278012 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:02.277913    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-7bb1c2/distinct-w3-7bb1c2-kserve-8d695596c-8zgtw" podStartSLOduration=1.277900257 podStartE2EDuration="1.277900257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:10:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:10:02.27472555 +0000 UTC m=+1179.879033114" watchObservedRunningTime="2026-10-30 22:10:02.277900257 +0000 UTC m=+1179.882207854"
Oct 30 22:10:02.601958 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:02.601881    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-7bb1c2/distinct-w3-7bb1c2-kserve-8d695596c-8zgtw"
Oct 30 22:10:10.574070 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:10:10.574031    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 22:10:11.601599 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:11.601566    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-7bb1c2/distinct-w3-7bb1c2-kserve-8d695596c-8zgtw"
Oct 30 22:10:11.613967 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:11.613942    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-7bb1c2/distinct-w3-7bb1c2-kserve-8d695596c-8zgtw"
Oct 30 22:10:12.303313 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:12.303270    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-7bb1c2/distinct-w3-7bb1c2-kserve-8d695596c-8zgtw"
Oct 30 22:10:31.737749 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:31.737713    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-f2qtx"]
Oct 30 22:10:31.743049 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:31.743027    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-f2qtx"
Oct 30 22:10:31.856029 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:31.855985    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95f62fb3-9a67-4bca-9483-8aedadf82cd6-credential-socket\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-f2qtx\" (UID: \"95f62fb3-9a67-4bca-9483-8aedadf82cd6\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-f2qtx"
Oct 30 22:10:31.856216 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:31.856033    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95f62fb3-9a67-4bca-9483-8aedadf82cd6-istiod-ca-cert\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-f2qtx\" (UID: \"95f62fb3-9a67-4bca-9483-8aedadf82cd6\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-f2qtx"
Oct 30 22:10:31.856216 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:31.856128    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5fgs\" (UniqueName: \"kubernetes.io/projected/95f62fb3-9a67-4bca-9483-8aedadf82cd6-kube-api-access-w5fgs\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-f2qtx\" (UID: \"95f62fb3-9a67-4bca-9483-8aedadf82cd6\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-f2qtx"
Oct 30 22:10:31.856331 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:31.856266    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95f62fb3-9a67-4bca-9483-8aedadf82cd6-istio-podinfo\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-f2qtx\" (UID: \"95f62fb3-9a67-4bca-9483-8aedadf82cd6\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-f2qtx"
Oct 30 22:10:31.856393 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:31.856329    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95f62fb3-9a67-4bca-9483-8aedadf82cd6-istio-envoy\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-f2qtx\" (UID: \"95f62fb3-9a67-4bca-9483-8aedadf82cd6\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-f2qtx"
Oct 30 22:10:31.856535 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:31.856519    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95f62fb3-9a67-4bca-9483-8aedadf82cd6-workload-certs\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-f2qtx\" (UID: \"95f62fb3-9a67-4bca-9483-8aedadf82cd6\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-f2qtx"
Oct 30 22:10:31.856595 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:31.856554    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95f62fb3-9a67-4bca-9483-8aedadf82cd6-workload-socket\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-f2qtx\" (UID: \"95f62fb3-9a67-4bca-9483-8aedadf82cd6\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-f2qtx"
Oct 30 22:10:31.856652 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:31.856601    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95f62fb3-9a67-4bca-9483-8aedadf82cd6-istio-data\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-f2qtx\" (UID: \"95f62fb3-9a67-4bca-9483-8aedadf82cd6\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-f2qtx"
Oct 30 22:10:31.856703 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:31.856647    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95f62fb3-9a67-4bca-9483-8aedadf82cd6-istio-token\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-f2qtx\" (UID: \"95f62fb3-9a67-4bca-9483-8aedadf82cd6\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-f2qtx"
Oct 30 22:10:31.941884 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:31.941806    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-f2qtx"]
Oct 30 22:10:31.958029 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:31.958002    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95f62fb3-9a67-4bca-9483-8aedadf82cd6-istio-data\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-f2qtx\" (UID: \"95f62fb3-9a67-4bca-9483-8aedadf82cd6\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-f2qtx"
Oct 30 22:10:31.958192 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:31.958054    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95f62fb3-9a67-4bca-9483-8aedadf82cd6-istio-token\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-f2qtx\" (UID: \"95f62fb3-9a67-4bca-9483-8aedadf82cd6\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-f2qtx"
Oct 30 22:10:31.958192 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:31.958122    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95f62fb3-9a67-4bca-9483-8aedadf82cd6-credential-socket\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-f2qtx\" (UID: \"95f62fb3-9a67-4bca-9483-8aedadf82cd6\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-f2qtx"
Oct 30 22:10:31.958341 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:31.958263    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95f62fb3-9a67-4bca-9483-8aedadf82cd6-istiod-ca-cert\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-f2qtx\" (UID: \"95f62fb3-9a67-4bca-9483-8aedadf82cd6\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-f2qtx"
Oct 30 22:10:31.958503 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:31.958481    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95f62fb3-9a67-4bca-9483-8aedadf82cd6-istio-data\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-f2qtx\" (UID: \"95f62fb3-9a67-4bca-9483-8aedadf82cd6\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-f2qtx"
Oct 30 22:10:31.958579 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:31.958563    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95f62fb3-9a67-4bca-9483-8aedadf82cd6-credential-socket\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-f2qtx\" (UID: \"95f62fb3-9a67-4bca-9483-8aedadf82cd6\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-f2qtx"
Oct 30 22:10:31.959019 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:31.958996    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95f62fb3-9a67-4bca-9483-8aedadf82cd6-istiod-ca-cert\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-f2qtx\" (UID: \"95f62fb3-9a67-4bca-9483-8aedadf82cd6\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-f2qtx"
Oct 30 22:10:31.959095 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:31.959075    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5fgs\" (UniqueName: \"kubernetes.io/projected/95f62fb3-9a67-4bca-9483-8aedadf82cd6-kube-api-access-w5fgs\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-f2qtx\" (UID: \"95f62fb3-9a67-4bca-9483-8aedadf82cd6\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-f2qtx"
Oct 30 22:10:31.959195 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:31.959177    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95f62fb3-9a67-4bca-9483-8aedadf82cd6-istio-podinfo\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-f2qtx\" (UID: \"95f62fb3-9a67-4bca-9483-8aedadf82cd6\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-f2qtx"
Oct 30 22:10:31.959273 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:31.959214    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95f62fb3-9a67-4bca-9483-8aedadf82cd6-istio-envoy\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-f2qtx\" (UID: \"95f62fb3-9a67-4bca-9483-8aedadf82cd6\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-f2qtx"
Oct 30 22:10:31.959390 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:31.959367    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95f62fb3-9a67-4bca-9483-8aedadf82cd6-workload-certs\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-f2qtx\" (UID: \"95f62fb3-9a67-4bca-9483-8aedadf82cd6\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-f2qtx"
Oct 30 22:10:31.959538 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:31.959426    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95f62fb3-9a67-4bca-9483-8aedadf82cd6-workload-socket\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-f2qtx\" (UID: \"95f62fb3-9a67-4bca-9483-8aedadf82cd6\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-f2qtx"
Oct 30 22:10:31.959730 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:31.959703    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95f62fb3-9a67-4bca-9483-8aedadf82cd6-workload-certs\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-f2qtx\" (UID: \"95f62fb3-9a67-4bca-9483-8aedadf82cd6\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-f2qtx"
Oct 30 22:10:31.959819 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:31.959787    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95f62fb3-9a67-4bca-9483-8aedadf82cd6-workload-socket\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-f2qtx\" (UID: \"95f62fb3-9a67-4bca-9483-8aedadf82cd6\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-f2qtx"
Oct 30 22:10:31.961613 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:31.961585    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95f62fb3-9a67-4bca-9483-8aedadf82cd6-istio-envoy\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-f2qtx\" (UID: \"95f62fb3-9a67-4bca-9483-8aedadf82cd6\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-f2qtx"
Oct 30 22:10:31.961714 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:31.961695    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95f62fb3-9a67-4bca-9483-8aedadf82cd6-istio-podinfo\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-f2qtx\" (UID: \"95f62fb3-9a67-4bca-9483-8aedadf82cd6\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-f2qtx"
Oct 30 22:10:32.016174 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:32.016110    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95f62fb3-9a67-4bca-9483-8aedadf82cd6-istio-token\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-f2qtx\" (UID: \"95f62fb3-9a67-4bca-9483-8aedadf82cd6\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-f2qtx"
Oct 30 22:10:32.016597 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:32.016577    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5fgs\" (UniqueName: \"kubernetes.io/projected/95f62fb3-9a67-4bca-9483-8aedadf82cd6-kube-api-access-w5fgs\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-f2qtx\" (UID: \"95f62fb3-9a67-4bca-9483-8aedadf82cd6\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-f2qtx"
Oct 30 22:10:32.056753 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:32.056730    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-f2qtx"
Oct 30 22:10:32.212490 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:32.212459    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9mpx4"]
Oct 30 22:10:32.217795 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:32.217773    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9mpx4"
Oct 30 22:10:32.278218 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:32.278147    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9mpx4"]
Oct 30 22:10:32.364772 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:32.364733    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/103020c5-ca66-48bf-8086-1c63af040de5-workload-socket\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9mpx4\" (UID: \"103020c5-ca66-48bf-8086-1c63af040de5\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9mpx4"
Oct 30 22:10:32.364938 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:32.364870    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/103020c5-ca66-48bf-8086-1c63af040de5-workload-certs\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9mpx4\" (UID: \"103020c5-ca66-48bf-8086-1c63af040de5\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9mpx4"
Oct 30 22:10:32.364982 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:32.364937    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/103020c5-ca66-48bf-8086-1c63af040de5-istio-envoy\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9mpx4\" (UID: \"103020c5-ca66-48bf-8086-1c63af040de5\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9mpx4"
Oct 30 22:10:32.365017 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:32.364981    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/103020c5-ca66-48bf-8086-1c63af040de5-istio-data\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9mpx4\" (UID: \"103020c5-ca66-48bf-8086-1c63af040de5\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9mpx4"
Oct 30 22:10:32.365059 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:32.365046    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/103020c5-ca66-48bf-8086-1c63af040de5-credential-socket\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9mpx4\" (UID: \"103020c5-ca66-48bf-8086-1c63af040de5\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9mpx4"
Oct 30 22:10:32.365179 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:32.365160    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64hrf\" (UniqueName: \"kubernetes.io/projected/103020c5-ca66-48bf-8086-1c63af040de5-kube-api-access-64hrf\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9mpx4\" (UID: \"103020c5-ca66-48bf-8086-1c63af040de5\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9mpx4"
Oct 30 22:10:32.365226 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:32.365198    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/103020c5-ca66-48bf-8086-1c63af040de5-istio-token\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9mpx4\" (UID: \"103020c5-ca66-48bf-8086-1c63af040de5\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9mpx4"
Oct 30 22:10:32.365307 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:32.365293    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/103020c5-ca66-48bf-8086-1c63af040de5-istio-podinfo\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9mpx4\" (UID: \"103020c5-ca66-48bf-8086-1c63af040de5\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9mpx4"
Oct 30 22:10:32.365368 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:32.365355    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/103020c5-ca66-48bf-8086-1c63af040de5-istiod-ca-cert\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9mpx4\" (UID: \"103020c5-ca66-48bf-8086-1c63af040de5\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9mpx4"
Oct 30 22:10:32.467072 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:32.467037    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64hrf\" (UniqueName: \"kubernetes.io/projected/103020c5-ca66-48bf-8086-1c63af040de5-kube-api-access-64hrf\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9mpx4\" (UID: \"103020c5-ca66-48bf-8086-1c63af040de5\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9mpx4"
Oct 30 22:10:32.467257 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:32.467081    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/103020c5-ca66-48bf-8086-1c63af040de5-istio-token\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9mpx4\" (UID: \"103020c5-ca66-48bf-8086-1c63af040de5\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9mpx4"
Oct 30 22:10:32.467326 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:32.467277    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/103020c5-ca66-48bf-8086-1c63af040de5-istio-podinfo\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9mpx4\" (UID: \"103020c5-ca66-48bf-8086-1c63af040de5\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9mpx4"
Oct 30 22:10:32.467383 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:32.467366    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/103020c5-ca66-48bf-8086-1c63af040de5-istiod-ca-cert\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9mpx4\" (UID: \"103020c5-ca66-48bf-8086-1c63af040de5\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9mpx4"
Oct 30 22:10:32.467483 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:32.467447    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/103020c5-ca66-48bf-8086-1c63af040de5-workload-socket\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9mpx4\" (UID: \"103020c5-ca66-48bf-8086-1c63af040de5\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9mpx4"
Oct 30 22:10:32.467558 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:32.467542    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/103020c5-ca66-48bf-8086-1c63af040de5-workload-certs\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9mpx4\" (UID: \"103020c5-ca66-48bf-8086-1c63af040de5\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9mpx4"
Oct 30 22:10:32.467621 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:32.467569    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/103020c5-ca66-48bf-8086-1c63af040de5-istio-envoy\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9mpx4\" (UID: \"103020c5-ca66-48bf-8086-1c63af040de5\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9mpx4"
Oct 30 22:10:32.467621 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:32.467607    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/103020c5-ca66-48bf-8086-1c63af040de5-istio-data\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9mpx4\" (UID: \"103020c5-ca66-48bf-8086-1c63af040de5\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9mpx4"
Oct 30 22:10:32.467720 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:32.467678    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/103020c5-ca66-48bf-8086-1c63af040de5-credential-socket\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9mpx4\" (UID: \"103020c5-ca66-48bf-8086-1c63af040de5\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9mpx4"
Oct 30 22:10:32.467937 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:32.467884    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/103020c5-ca66-48bf-8086-1c63af040de5-workload-socket\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9mpx4\" (UID: \"103020c5-ca66-48bf-8086-1c63af040de5\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9mpx4"
Oct 30 22:10:32.468059 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:32.467949    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/103020c5-ca66-48bf-8086-1c63af040de5-workload-certs\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9mpx4\" (UID: \"103020c5-ca66-48bf-8086-1c63af040de5\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9mpx4"
Oct 30 22:10:32.468120 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:32.468072    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/103020c5-ca66-48bf-8086-1c63af040de5-credential-socket\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9mpx4\" (UID: \"103020c5-ca66-48bf-8086-1c63af040de5\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9mpx4"
Oct 30 22:10:32.468120 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:32.468102    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/103020c5-ca66-48bf-8086-1c63af040de5-istio-data\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9mpx4\" (UID: \"103020c5-ca66-48bf-8086-1c63af040de5\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9mpx4"
Oct 30 22:10:32.468193 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:32.468143    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/103020c5-ca66-48bf-8086-1c63af040de5-istiod-ca-cert\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9mpx4\" (UID: \"103020c5-ca66-48bf-8086-1c63af040de5\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9mpx4"
Oct 30 22:10:32.469718 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:32.469696    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/103020c5-ca66-48bf-8086-1c63af040de5-istio-envoy\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9mpx4\" (UID: \"103020c5-ca66-48bf-8086-1c63af040de5\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9mpx4"
Oct 30 22:10:32.469837 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:32.469819    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/103020c5-ca66-48bf-8086-1c63af040de5-istio-podinfo\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9mpx4\" (UID: \"103020c5-ca66-48bf-8086-1c63af040de5\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9mpx4"
Oct 30 22:10:32.506512 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:32.506485    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-f2qtx"]
Oct 30 22:10:32.507275 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:10:32.507241    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95f62fb3_9a67_4bca_9483_8aedadf82cd6.slice/crio-c230e28df08b368fcbdebf557e259c7b9ca4e8bba59d2923d9395550b96560c9 WatchSource:0}: Error finding container c230e28df08b368fcbdebf557e259c7b9ca4e8bba59d2923d9395550b96560c9: Status 404 returned error can't find the container with id c230e28df08b368fcbdebf557e259c7b9ca4e8bba59d2923d9395550b96560c9
Oct 30 22:10:32.509697 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:32.509666    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:10:32.509754 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:32.509731    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:10:32.509802 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:32.509762    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:10:32.526151 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:32.526132    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/103020c5-ca66-48bf-8086-1c63af040de5-istio-token\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9mpx4\" (UID: \"103020c5-ca66-48bf-8086-1c63af040de5\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9mpx4"
Oct 30 22:10:32.526255 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:32.526235    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64hrf\" (UniqueName: \"kubernetes.io/projected/103020c5-ca66-48bf-8086-1c63af040de5-kube-api-access-64hrf\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9mpx4\" (UID: \"103020c5-ca66-48bf-8086-1c63af040de5\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9mpx4"
Oct 30 22:10:32.531601 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:32.531558    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9mpx4"
Oct 30 22:10:32.976652 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:32.976630    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9mpx4"]
Oct 30 22:10:32.978255 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:10:32.978219    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod103020c5_ca66_48bf_8086_1c63af040de5.slice/crio-ebccdb4490e2ffbb7c440fdad951ceac418b9ee654bf2574aa088501baed65f3 WatchSource:0}: Error finding container ebccdb4490e2ffbb7c440fdad951ceac418b9ee654bf2574aa088501baed65f3: Status 404 returned error can't find the container with id ebccdb4490e2ffbb7c440fdad951ceac418b9ee654bf2574aa088501baed65f3
Oct 30 22:10:32.981679 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:32.981643    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:10:32.981760 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:32.981728    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:10:32.981800 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:32.981769    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:10:33.413930 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:33.413886    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9mpx4" event={"ID":"103020c5-ca66-48bf-8086-1c63af040de5","Type":"ContainerStarted","Data":"90509b4a23b32df1625c3733730077caae1a914d1b7a458a659c8c59481bfe90"}
Oct 30 22:10:33.414116 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:33.413949    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9mpx4" event={"ID":"103020c5-ca66-48bf-8086-1c63af040de5","Type":"ContainerStarted","Data":"ebccdb4490e2ffbb7c440fdad951ceac418b9ee654bf2574aa088501baed65f3"}
Oct 30 22:10:33.415831 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:33.415798    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-f2qtx" event={"ID":"95f62fb3-9a67-4bca-9483-8aedadf82cd6","Type":"ContainerStarted","Data":"6a9c06c3d44802378cdacb2dea0eb91d1a51c076fa1781a621dbffeb2e301866"}
Oct 30 22:10:33.415985 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:33.415844    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-f2qtx" event={"ID":"95f62fb3-9a67-4bca-9483-8aedadf82cd6","Type":"ContainerStarted","Data":"c230e28df08b368fcbdebf557e259c7b9ca4e8bba59d2923d9395550b96560c9"}
Oct 30 22:10:33.543422 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:33.543032    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9mpx4"
Oct 30 22:10:33.579964 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:33.579898    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9mpx4" podStartSLOduration=1.5798845419999998 podStartE2EDuration="1.579884542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:10:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:10:33.577086649 +0000 UTC m=+1211.181394201" watchObservedRunningTime="2026-10-30 22:10:33.579884542 +0000 UTC m=+1211.184192094"
Oct 30 22:10:33.697529 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:33.697428    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-f2qtx" podStartSLOduration=2.697413648 podStartE2EDuration="2.697413648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:10:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:10:33.695593181 +0000 UTC m=+1211.299900757" watchObservedRunningTime="2026-10-30 22:10:33.697413648 +0000 UTC m=+1211.301721192"
Oct 30 22:10:34.062281 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:34.062142    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-f2qtx"
Oct 30 22:10:34.062914 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:34.062889    5678 patch_prober.go:29] interesting pod/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-f2qtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Oct 30 22:10:34.063079 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:34.063053    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-f2qtx" podUID="95f62fb3-9a67-4bca-9483-8aedadf82cd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Oct 30 22:10:34.549426 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:34.546601    5678 patch_prober.go:29] interesting pod/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9mpx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Oct 30 22:10:34.549426 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:34.546662    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9mpx4" podUID="103020c5-ca66-48bf-8086-1c63af040de5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Oct 30 22:10:35.063811 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:35.063774    5678 patch_prober.go:29] interesting pod/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-f2qtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Oct 30 22:10:35.064817 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:35.064786    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-f2qtx" podUID="95f62fb3-9a67-4bca-9483-8aedadf82cd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Oct 30 22:10:35.541426 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:35.540371    5678 patch_prober.go:29] interesting pod/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9mpx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Oct 30 22:10:35.541426 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:35.540461    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9mpx4" podUID="103020c5-ca66-48bf-8086-1c63af040de5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Oct 30 22:10:36.058827 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:36.058790    5678 patch_prober.go:29] interesting pod/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-f2qtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Oct 30 22:10:36.059136 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:36.059103    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-f2qtx" podUID="95f62fb3-9a67-4bca-9483-8aedadf82cd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Oct 30 22:10:36.559424 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:36.540636    5678 patch_prober.go:29] interesting pod/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9mpx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Oct 30 22:10:36.561700 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:36.561660    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9mpx4" podUID="103020c5-ca66-48bf-8086-1c63af040de5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Oct 30 22:10:37.057864 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:37.057828    5678 patch_prober.go:29] interesting pod/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-f2qtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Oct 30 22:10:37.058188 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:37.058148    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-f2qtx" podUID="95f62fb3-9a67-4bca-9483-8aedadf82cd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Oct 30 22:10:37.534129 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:37.533696    5678 patch_prober.go:29] interesting pod/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9mpx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Oct 30 22:10:37.534129 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:37.533752    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9mpx4" podUID="103020c5-ca66-48bf-8086-1c63af040de5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Oct 30 22:10:38.077537 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:38.077494    5678 patch_prober.go:29] interesting pod/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-f2qtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Oct 30 22:10:38.078155 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:38.078120    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-f2qtx" podUID="95f62fb3-9a67-4bca-9483-8aedadf82cd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Oct 30 22:10:38.587426 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:38.569563    5678 patch_prober.go:29] interesting pod/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9mpx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Oct 30 22:10:38.587426 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:38.569642    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9mpx4" podUID="103020c5-ca66-48bf-8086-1c63af040de5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Oct 30 22:10:39.064423 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:39.063862    5678 patch_prober.go:29] interesting pod/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-f2qtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Oct 30 22:10:39.064423 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:39.063941    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-f2qtx" podUID="95f62fb3-9a67-4bca-9483-8aedadf82cd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Oct 30 22:10:39.275423 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:39.275264    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7bb1c2/distinct-w3-7bb1c2-kserve-8d695596c-8zgtw" podUID="36f0a6a2-54be-49bf-9202-502960cbb134" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded"
Oct 30 22:10:39.474426 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:39.473489    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7bb1c2/premium-w3-7bb1c2-kserve-7dbf875d88-dqxn5" podUID="a58c911f-e2eb-4820-a6b6-24afe78b5fd4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:10:39.496529 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:39.496483    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9ndd9" podUID="81bd6415-f42f-446b-a64e-956f794b3956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": context deadline exceeded"
Oct 30 22:10:39.532955 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:39.532924    5678 patch_prober.go:29] interesting pod/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9mpx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Oct 30 22:10:39.533219 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:39.533194    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9mpx4" podUID="103020c5-ca66-48bf-8086-1c63af040de5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Oct 30 22:10:39.570812 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:39.570767    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-65e59f/premium-w4-65e59f-kserve-6dc49468f8-rlzhx" podUID="8590a783-eb24-4759-ad74-1369f36aa9d2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:10:39.837478 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:39.831494    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb0b93/simulator-w6-fb0b93-kserve-55b784b479-588bn" podUID="951a6f7e-a22e-40f2-ace2-1ce5d3bc9b49" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 30 22:10:40.049947 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:40.049897    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-65e59f/simulator-w4-65e59f-kserve-85698bcd47-przch" podUID="ee5e843b-17be-4623-8c95-dc07c8b7f506" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:10:40.060792 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:40.060759    5678 patch_prober.go:29] interesting pod/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-f2qtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Oct 30 22:10:40.061049 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:40.061025    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-f2qtx" podUID="95f62fb3-9a67-4bca-9483-8aedadf82cd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Oct 30 22:10:40.531992 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:40.531955    5678 patch_prober.go:29] interesting pod/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9mpx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Oct 30 22:10:40.532428 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:40.532032    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9mpx4" podUID="103020c5-ca66-48bf-8086-1c63af040de5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Oct 30 22:10:40.587951 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:10:40.587915    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 22:10:41.074694 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:41.074659    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-f2qtx"
Oct 30 22:10:41.075238 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:41.075021    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-f2qtx"
Oct 30 22:10:41.076304 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:41.076242    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-f2qtx"
Oct 30 22:10:41.536268 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:41.536235    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9mpx4"
Oct 30 22:10:41.536756 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:41.536682    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9mpx4"
Oct 30 22:10:41.537366 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:10:41.537346    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9mpx4"
Oct 30 22:11:00.983276 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:00.983246    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cqvp9"]
Oct 30 22:11:00.987921 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:00.987898    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cqvp9"
Oct 30 22:11:01.057904 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:01.057853    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e913a61-bea1-4281-9d5d-cdab5506a864-istio-envoy\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cqvp9\" (UID: \"4e913a61-bea1-4281-9d5d-cdab5506a864\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cqvp9"
Oct 30 22:11:01.057904 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:01.057903    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hczps\" (UniqueName: \"kubernetes.io/projected/4e913a61-bea1-4281-9d5d-cdab5506a864-kube-api-access-hczps\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cqvp9\" (UID: \"4e913a61-bea1-4281-9d5d-cdab5506a864\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cqvp9"
Oct 30 22:11:01.058147 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:01.058085    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e913a61-bea1-4281-9d5d-cdab5506a864-workload-socket\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cqvp9\" (UID: \"4e913a61-bea1-4281-9d5d-cdab5506a864\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cqvp9"
Oct 30 22:11:01.058147 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:01.058112    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e913a61-bea1-4281-9d5d-cdab5506a864-istiod-ca-cert\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cqvp9\" (UID: \"4e913a61-bea1-4281-9d5d-cdab5506a864\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cqvp9"
Oct 30 22:11:01.058147 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:01.058135    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e913a61-bea1-4281-9d5d-cdab5506a864-istio-data\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cqvp9\" (UID: \"4e913a61-bea1-4281-9d5d-cdab5506a864\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cqvp9"
Oct 30 22:11:01.058313 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:01.058176    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e913a61-bea1-4281-9d5d-cdab5506a864-istio-token\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cqvp9\" (UID: \"4e913a61-bea1-4281-9d5d-cdab5506a864\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cqvp9"
Oct 30 22:11:01.058393 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:01.058371    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e913a61-bea1-4281-9d5d-cdab5506a864-credential-socket\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cqvp9\" (UID: \"4e913a61-bea1-4281-9d5d-cdab5506a864\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cqvp9"
Oct 30 22:11:01.058497 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:01.058471    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e913a61-bea1-4281-9d5d-cdab5506a864-workload-certs\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cqvp9\" (UID: \"4e913a61-bea1-4281-9d5d-cdab5506a864\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cqvp9"
Oct 30 22:11:01.058497 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:01.058493    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e913a61-bea1-4281-9d5d-cdab5506a864-istio-podinfo\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cqvp9\" (UID: \"4e913a61-bea1-4281-9d5d-cdab5506a864\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cqvp9"
Oct 30 22:11:01.103163 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:01.103130    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cqvp9"]
Oct 30 22:11:01.159350 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:01.159311    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e913a61-bea1-4281-9d5d-cdab5506a864-credential-socket\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cqvp9\" (UID: \"4e913a61-bea1-4281-9d5d-cdab5506a864\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cqvp9"
Oct 30 22:11:01.159552 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:01.159394    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e913a61-bea1-4281-9d5d-cdab5506a864-workload-certs\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cqvp9\" (UID: \"4e913a61-bea1-4281-9d5d-cdab5506a864\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cqvp9"
Oct 30 22:11:01.159552 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:01.159443    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e913a61-bea1-4281-9d5d-cdab5506a864-istio-podinfo\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cqvp9\" (UID: \"4e913a61-bea1-4281-9d5d-cdab5506a864\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cqvp9"
Oct 30 22:11:01.159552 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:01.159549    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e913a61-bea1-4281-9d5d-cdab5506a864-istio-envoy\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cqvp9\" (UID: \"4e913a61-bea1-4281-9d5d-cdab5506a864\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cqvp9"
Oct 30 22:11:01.159729 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:01.159574    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hczps\" (UniqueName: \"kubernetes.io/projected/4e913a61-bea1-4281-9d5d-cdab5506a864-kube-api-access-hczps\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cqvp9\" (UID: \"4e913a61-bea1-4281-9d5d-cdab5506a864\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cqvp9"
Oct 30 22:11:01.159729 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:01.159680    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e913a61-bea1-4281-9d5d-cdab5506a864-workload-socket\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cqvp9\" (UID: \"4e913a61-bea1-4281-9d5d-cdab5506a864\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cqvp9"
Oct 30 22:11:01.159729 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:01.159703    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e913a61-bea1-4281-9d5d-cdab5506a864-istiod-ca-cert\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cqvp9\" (UID: \"4e913a61-bea1-4281-9d5d-cdab5506a864\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cqvp9"
Oct 30 22:11:01.159898 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:01.159728    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e913a61-bea1-4281-9d5d-cdab5506a864-credential-socket\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cqvp9\" (UID: \"4e913a61-bea1-4281-9d5d-cdab5506a864\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cqvp9"
Oct 30 22:11:01.159898 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:01.159864    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e913a61-bea1-4281-9d5d-cdab5506a864-workload-certs\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cqvp9\" (UID: \"4e913a61-bea1-4281-9d5d-cdab5506a864\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cqvp9"
Oct 30 22:11:01.160094 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:01.159732    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e913a61-bea1-4281-9d5d-cdab5506a864-istio-data\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cqvp9\" (UID: \"4e913a61-bea1-4281-9d5d-cdab5506a864\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cqvp9"
Oct 30 22:11:01.160262 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:01.160240    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e913a61-bea1-4281-9d5d-cdab5506a864-workload-socket\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cqvp9\" (UID: \"4e913a61-bea1-4281-9d5d-cdab5506a864\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cqvp9"
Oct 30 22:11:01.160366 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:01.160344    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e913a61-bea1-4281-9d5d-cdab5506a864-istio-token\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cqvp9\" (UID: \"4e913a61-bea1-4281-9d5d-cdab5506a864\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cqvp9"
Oct 30 22:11:01.160685 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:01.160654    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e913a61-bea1-4281-9d5d-cdab5506a864-istio-data\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cqvp9\" (UID: \"4e913a61-bea1-4281-9d5d-cdab5506a864\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cqvp9"
Oct 30 22:11:01.161355 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:01.161332    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e913a61-bea1-4281-9d5d-cdab5506a864-istiod-ca-cert\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cqvp9\" (UID: \"4e913a61-bea1-4281-9d5d-cdab5506a864\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cqvp9"
Oct 30 22:11:01.162838 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:01.162816    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e913a61-bea1-4281-9d5d-cdab5506a864-istio-podinfo\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cqvp9\" (UID: \"4e913a61-bea1-4281-9d5d-cdab5506a864\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cqvp9"
Oct 30 22:11:01.162945 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:01.162851    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e913a61-bea1-4281-9d5d-cdab5506a864-istio-envoy\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cqvp9\" (UID: \"4e913a61-bea1-4281-9d5d-cdab5506a864\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cqvp9"
Oct 30 22:11:01.218375 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:01.218343    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e913a61-bea1-4281-9d5d-cdab5506a864-istio-token\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cqvp9\" (UID: \"4e913a61-bea1-4281-9d5d-cdab5506a864\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cqvp9"
Oct 30 22:11:01.218518 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:01.218435    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hczps\" (UniqueName: \"kubernetes.io/projected/4e913a61-bea1-4281-9d5d-cdab5506a864-kube-api-access-hczps\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cqvp9\" (UID: \"4e913a61-bea1-4281-9d5d-cdab5506a864\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cqvp9"
Oct 30 22:11:01.301543 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:01.301448    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cqvp9"
Oct 30 22:11:01.687596 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:01.687556    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-l8b5h"]
Oct 30 22:11:01.692924 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:01.692900    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-l8b5h"
Oct 30 22:11:01.768699 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:01.768663    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09fbaa51-372b-406e-a478-3a7d52a8a618-istio-podinfo\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-l8b5h\" (UID: \"09fbaa51-372b-406e-a478-3a7d52a8a618\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-l8b5h"
Oct 30 22:11:01.768906 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:01.768716    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09fbaa51-372b-406e-a478-3a7d52a8a618-istiod-ca-cert\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-l8b5h\" (UID: \"09fbaa51-372b-406e-a478-3a7d52a8a618\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-l8b5h"
Oct 30 22:11:01.768950 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:01.768902    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09fbaa51-372b-406e-a478-3a7d52a8a618-istio-envoy\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-l8b5h\" (UID: \"09fbaa51-372b-406e-a478-3a7d52a8a618\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-l8b5h"
Oct 30 22:11:01.769006 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:01.768957    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09fbaa51-372b-406e-a478-3a7d52a8a618-istio-data\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-l8b5h\" (UID: \"09fbaa51-372b-406e-a478-3a7d52a8a618\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-l8b5h"
Oct 30 22:11:01.769096 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:01.769075    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09fbaa51-372b-406e-a478-3a7d52a8a618-workload-certs\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-l8b5h\" (UID: \"09fbaa51-372b-406e-a478-3a7d52a8a618\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-l8b5h"
Oct 30 22:11:01.769185 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:01.769169    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09fbaa51-372b-406e-a478-3a7d52a8a618-credential-socket\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-l8b5h\" (UID: \"09fbaa51-372b-406e-a478-3a7d52a8a618\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-l8b5h"
Oct 30 22:11:01.769271 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:01.769256    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09fbaa51-372b-406e-a478-3a7d52a8a618-workload-socket\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-l8b5h\" (UID: \"09fbaa51-372b-406e-a478-3a7d52a8a618\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-l8b5h"
Oct 30 22:11:01.769325 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:01.769300    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09fbaa51-372b-406e-a478-3a7d52a8a618-istio-token\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-l8b5h\" (UID: \"09fbaa51-372b-406e-a478-3a7d52a8a618\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-l8b5h"
Oct 30 22:11:01.769372 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:01.769338    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fv2sh\" (UniqueName: \"kubernetes.io/projected/09fbaa51-372b-406e-a478-3a7d52a8a618-kube-api-access-fv2sh\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-l8b5h\" (UID: \"09fbaa51-372b-406e-a478-3a7d52a8a618\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-l8b5h"
Oct 30 22:11:01.789703 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:01.789661    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cqvp9"]
Oct 30 22:11:01.793674 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:01.793630    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:11:01.793775 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:01.793721    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:11:01.793775 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:01.793765    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:11:01.838380 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:01.838344    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-l8b5h"]
Oct 30 22:11:01.847677 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:01.847621    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cqvp9" event={"ID":"4e913a61-bea1-4281-9d5d-cdab5506a864","Type":"ContainerStarted","Data":"2472b400321d9a70c466ba473f4c76c48057ea4e6c843f2f25bd74375c3174f1"}
Oct 30 22:11:01.871105 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:01.871068    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09fbaa51-372b-406e-a478-3a7d52a8a618-istio-podinfo\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-l8b5h\" (UID: \"09fbaa51-372b-406e-a478-3a7d52a8a618\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-l8b5h"
Oct 30 22:11:01.871229 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:01.871134    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09fbaa51-372b-406e-a478-3a7d52a8a618-istiod-ca-cert\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-l8b5h\" (UID: \"09fbaa51-372b-406e-a478-3a7d52a8a618\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-l8b5h"
Oct 30 22:11:01.871295 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:01.871231    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09fbaa51-372b-406e-a478-3a7d52a8a618-istio-envoy\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-l8b5h\" (UID: \"09fbaa51-372b-406e-a478-3a7d52a8a618\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-l8b5h"
Oct 30 22:11:01.871295 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:01.871271    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09fbaa51-372b-406e-a478-3a7d52a8a618-istio-data\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-l8b5h\" (UID: \"09fbaa51-372b-406e-a478-3a7d52a8a618\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-l8b5h"
Oct 30 22:11:01.871443 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:01.871321    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09fbaa51-372b-406e-a478-3a7d52a8a618-workload-certs\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-l8b5h\" (UID: \"09fbaa51-372b-406e-a478-3a7d52a8a618\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-l8b5h"
Oct 30 22:11:01.871443 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:01.871383    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09fbaa51-372b-406e-a478-3a7d52a8a618-credential-socket\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-l8b5h\" (UID: \"09fbaa51-372b-406e-a478-3a7d52a8a618\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-l8b5h"
Oct 30 22:11:01.871565 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:01.871464    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09fbaa51-372b-406e-a478-3a7d52a8a618-workload-socket\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-l8b5h\" (UID: \"09fbaa51-372b-406e-a478-3a7d52a8a618\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-l8b5h"
Oct 30 22:11:01.871565 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:01.871506    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09fbaa51-372b-406e-a478-3a7d52a8a618-istio-token\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-l8b5h\" (UID: \"09fbaa51-372b-406e-a478-3a7d52a8a618\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-l8b5h"
Oct 30 22:11:01.871565 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:01.871546    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fv2sh\" (UniqueName: \"kubernetes.io/projected/09fbaa51-372b-406e-a478-3a7d52a8a618-kube-api-access-fv2sh\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-l8b5h\" (UID: \"09fbaa51-372b-406e-a478-3a7d52a8a618\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-l8b5h"
Oct 30 22:11:01.871991 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:01.871966    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09fbaa51-372b-406e-a478-3a7d52a8a618-istiod-ca-cert\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-l8b5h\" (UID: \"09fbaa51-372b-406e-a478-3a7d52a8a618\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-l8b5h"
Oct 30 22:11:01.872080 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:01.871995    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09fbaa51-372b-406e-a478-3a7d52a8a618-workload-certs\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-l8b5h\" (UID: \"09fbaa51-372b-406e-a478-3a7d52a8a618\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-l8b5h"
Oct 30 22:11:01.872215 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:01.872191    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09fbaa51-372b-406e-a478-3a7d52a8a618-workload-socket\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-l8b5h\" (UID: \"09fbaa51-372b-406e-a478-3a7d52a8a618\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-l8b5h"
Oct 30 22:11:01.872283 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:01.872244    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09fbaa51-372b-406e-a478-3a7d52a8a618-credential-socket\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-l8b5h\" (UID: \"09fbaa51-372b-406e-a478-3a7d52a8a618\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-l8b5h"
Oct 30 22:11:01.872544 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:01.872518    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09fbaa51-372b-406e-a478-3a7d52a8a618-istio-data\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-l8b5h\" (UID: \"09fbaa51-372b-406e-a478-3a7d52a8a618\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-l8b5h"
Oct 30 22:11:01.873775 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:01.873755    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09fbaa51-372b-406e-a478-3a7d52a8a618-istio-podinfo\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-l8b5h\" (UID: \"09fbaa51-372b-406e-a478-3a7d52a8a618\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-l8b5h"
Oct 30 22:11:01.873936 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:01.873920    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09fbaa51-372b-406e-a478-3a7d52a8a618-istio-envoy\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-l8b5h\" (UID: \"09fbaa51-372b-406e-a478-3a7d52a8a618\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-l8b5h"
Oct 30 22:11:01.932332 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:01.932291    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09fbaa51-372b-406e-a478-3a7d52a8a618-istio-token\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-l8b5h\" (UID: \"09fbaa51-372b-406e-a478-3a7d52a8a618\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-l8b5h"
Oct 30 22:11:01.932513 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:01.932490    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fv2sh\" (UniqueName: \"kubernetes.io/projected/09fbaa51-372b-406e-a478-3a7d52a8a618-kube-api-access-fv2sh\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-l8b5h\" (UID: \"09fbaa51-372b-406e-a478-3a7d52a8a618\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-l8b5h"
Oct 30 22:11:02.007075 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:02.006996    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-l8b5h"
Oct 30 22:11:02.452412 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:11:02.452359    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09fbaa51_372b_406e_a478_3a7d52a8a618.slice/crio-1e723b77f17c825c0303143b2634e378c6dba82165ab2b3a38a3d5a7398a5ab4 WatchSource:0}: Error finding container 1e723b77f17c825c0303143b2634e378c6dba82165ab2b3a38a3d5a7398a5ab4: Status 404 returned error can't find the container with id 1e723b77f17c825c0303143b2634e378c6dba82165ab2b3a38a3d5a7398a5ab4
Oct 30 22:11:02.452763 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:02.452743    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-l8b5h"]
Oct 30 22:11:02.455556 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:02.455526    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:11:02.455634 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:02.455590    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:11:02.455634 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:02.455619    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:11:02.706589 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:02.706513    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-2d6zs"]
Oct 30 22:11:02.711369 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:02.711346    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-2d6zs"
Oct 30 22:11:02.779960 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:02.779556    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-2d6zs"]
Oct 30 22:11:02.783203 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:02.783171    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8061c232-3802-4de3-b53c-52238ba2bc75-istio-envoy\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-2d6zs\" (UID: \"8061c232-3802-4de3-b53c-52238ba2bc75\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-2d6zs"
Oct 30 22:11:02.783334 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:02.783246    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8061c232-3802-4de3-b53c-52238ba2bc75-workload-certs\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-2d6zs\" (UID: \"8061c232-3802-4de3-b53c-52238ba2bc75\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-2d6zs"
Oct 30 22:11:02.783429 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:02.783393    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8061c232-3802-4de3-b53c-52238ba2bc75-credential-socket\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-2d6zs\" (UID: \"8061c232-3802-4de3-b53c-52238ba2bc75\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-2d6zs"
Oct 30 22:11:02.783545 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:02.783485    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8061c232-3802-4de3-b53c-52238ba2bc75-istiod-ca-cert\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-2d6zs\" (UID: \"8061c232-3802-4de3-b53c-52238ba2bc75\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-2d6zs"
Oct 30 22:11:02.783948 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:02.783665    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8061c232-3802-4de3-b53c-52238ba2bc75-istio-podinfo\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-2d6zs\" (UID: \"8061c232-3802-4de3-b53c-52238ba2bc75\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-2d6zs"
Oct 30 22:11:02.783948 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:02.783731    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8061c232-3802-4de3-b53c-52238ba2bc75-istio-token\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-2d6zs\" (UID: \"8061c232-3802-4de3-b53c-52238ba2bc75\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-2d6zs"
Oct 30 22:11:02.783948 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:02.783756    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jb2pz\" (UniqueName: \"kubernetes.io/projected/8061c232-3802-4de3-b53c-52238ba2bc75-kube-api-access-jb2pz\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-2d6zs\" (UID: \"8061c232-3802-4de3-b53c-52238ba2bc75\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-2d6zs"
Oct 30 22:11:02.783948 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:02.783801    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8061c232-3802-4de3-b53c-52238ba2bc75-istio-data\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-2d6zs\" (UID: \"8061c232-3802-4de3-b53c-52238ba2bc75\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-2d6zs"
Oct 30 22:11:02.783948 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:02.783855    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8061c232-3802-4de3-b53c-52238ba2bc75-workload-socket\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-2d6zs\" (UID: \"8061c232-3802-4de3-b53c-52238ba2bc75\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-2d6zs"
Oct 30 22:11:02.863051 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:02.863015    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cqvp9" event={"ID":"4e913a61-bea1-4281-9d5d-cdab5506a864","Type":"ContainerStarted","Data":"2f6d42261c7f0ce6f5c924ba6332ac7eb03692406eb0200a79b0b374ef17d304"}
Oct 30 22:11:02.865052 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:02.865020    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-l8b5h" event={"ID":"09fbaa51-372b-406e-a478-3a7d52a8a618","Type":"ContainerStarted","Data":"79803bb8b68800167f26faf4dc73405ced63ea2b0300cc93776be9cec8f69d84"}
Oct 30 22:11:02.865176 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:02.865078    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-l8b5h" event={"ID":"09fbaa51-372b-406e-a478-3a7d52a8a618","Type":"ContainerStarted","Data":"1e723b77f17c825c0303143b2634e378c6dba82165ab2b3a38a3d5a7398a5ab4"}
Oct 30 22:11:02.885166 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:02.885136    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8061c232-3802-4de3-b53c-52238ba2bc75-credential-socket\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-2d6zs\" (UID: \"8061c232-3802-4de3-b53c-52238ba2bc75\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-2d6zs"
Oct 30 22:11:02.885271 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:02.885243    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8061c232-3802-4de3-b53c-52238ba2bc75-istiod-ca-cert\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-2d6zs\" (UID: \"8061c232-3802-4de3-b53c-52238ba2bc75\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-2d6zs"
Oct 30 22:11:02.885389 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:02.885372    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8061c232-3802-4de3-b53c-52238ba2bc75-istio-podinfo\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-2d6zs\" (UID: \"8061c232-3802-4de3-b53c-52238ba2bc75\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-2d6zs"
Oct 30 22:11:02.885503 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:02.885469    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8061c232-3802-4de3-b53c-52238ba2bc75-istio-token\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-2d6zs\" (UID: \"8061c232-3802-4de3-b53c-52238ba2bc75\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-2d6zs"
Oct 30 22:11:02.885503 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:02.885497    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jb2pz\" (UniqueName: \"kubernetes.io/projected/8061c232-3802-4de3-b53c-52238ba2bc75-kube-api-access-jb2pz\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-2d6zs\" (UID: \"8061c232-3802-4de3-b53c-52238ba2bc75\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-2d6zs"
Oct 30 22:11:02.885615 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:02.885568    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8061c232-3802-4de3-b53c-52238ba2bc75-credential-socket\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-2d6zs\" (UID: \"8061c232-3802-4de3-b53c-52238ba2bc75\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-2d6zs"
Oct 30 22:11:02.886321 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:02.885572    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8061c232-3802-4de3-b53c-52238ba2bc75-istio-data\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-2d6zs\" (UID: \"8061c232-3802-4de3-b53c-52238ba2bc75\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-2d6zs"
Oct 30 22:11:02.886321 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:02.885790    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8061c232-3802-4de3-b53c-52238ba2bc75-workload-socket\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-2d6zs\" (UID: \"8061c232-3802-4de3-b53c-52238ba2bc75\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-2d6zs"
Oct 30 22:11:02.886321 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:02.885818    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8061c232-3802-4de3-b53c-52238ba2bc75-istio-data\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-2d6zs\" (UID: \"8061c232-3802-4de3-b53c-52238ba2bc75\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-2d6zs"
Oct 30 22:11:02.886321 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:02.885906    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8061c232-3802-4de3-b53c-52238ba2bc75-istio-envoy\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-2d6zs\" (UID: \"8061c232-3802-4de3-b53c-52238ba2bc75\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-2d6zs"
Oct 30 22:11:02.886321 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:02.885974    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8061c232-3802-4de3-b53c-52238ba2bc75-workload-certs\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-2d6zs\" (UID: \"8061c232-3802-4de3-b53c-52238ba2bc75\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-2d6zs"
Oct 30 22:11:02.886321 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:02.886029    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8061c232-3802-4de3-b53c-52238ba2bc75-istiod-ca-cert\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-2d6zs\" (UID: \"8061c232-3802-4de3-b53c-52238ba2bc75\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-2d6zs"
Oct 30 22:11:02.886321 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:02.886088    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8061c232-3802-4de3-b53c-52238ba2bc75-workload-socket\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-2d6zs\" (UID: \"8061c232-3802-4de3-b53c-52238ba2bc75\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-2d6zs"
Oct 30 22:11:02.886321 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:02.886281    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8061c232-3802-4de3-b53c-52238ba2bc75-workload-certs\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-2d6zs\" (UID: \"8061c232-3802-4de3-b53c-52238ba2bc75\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-2d6zs"
Oct 30 22:11:02.888199 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:02.888173    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8061c232-3802-4de3-b53c-52238ba2bc75-istio-podinfo\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-2d6zs\" (UID: \"8061c232-3802-4de3-b53c-52238ba2bc75\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-2d6zs"
Oct 30 22:11:02.888296 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:02.888236    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8061c232-3802-4de3-b53c-52238ba2bc75-istio-envoy\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-2d6zs\" (UID: \"8061c232-3802-4de3-b53c-52238ba2bc75\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-2d6zs"
Oct 30 22:11:02.953878 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:02.953850    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8061c232-3802-4de3-b53c-52238ba2bc75-istio-token\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-2d6zs\" (UID: \"8061c232-3802-4de3-b53c-52238ba2bc75\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-2d6zs"
Oct 30 22:11:02.955074 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:02.955046    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jb2pz\" (UniqueName: \"kubernetes.io/projected/8061c232-3802-4de3-b53c-52238ba2bc75-kube-api-access-jb2pz\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-2d6zs\" (UID: \"8061c232-3802-4de3-b53c-52238ba2bc75\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-2d6zs"
Oct 30 22:11:02.982238 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:02.982150    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cqvp9" podStartSLOduration=2.982134342 podStartE2EDuration="2.982134342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:11:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:11:02.97922883 +0000 UTC m=+1240.583536385" watchObservedRunningTime="2026-10-30 22:11:02.982134342 +0000 UTC m=+1240.586441895"
Oct 30 22:11:03.008057 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:03.008029    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-l8b5h"
Oct 30 22:11:03.009099 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:03.009079    5678 patch_prober.go:29] interesting pod/e2e-worker-w2-bc714d-openshift-default-67558b95cf-l8b5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Oct 30 22:11:03.009151 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:03.009122    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-l8b5h" podUID="09fbaa51-372b-406e-a478-3a7d52a8a618" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Oct 30 22:11:03.047040 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:03.047008    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-2d6zs"
Oct 30 22:11:03.104017 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:03.103956    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-l8b5h" podStartSLOduration=2.103937025 podStartE2EDuration="2.103937025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:11:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:11:03.101619805 +0000 UTC m=+1240.705927391" watchObservedRunningTime="2026-10-30 22:11:03.103937025 +0000 UTC m=+1240.708244580"
Oct 30 22:11:03.302595 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:03.302512    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cqvp9"
Oct 30 22:11:03.304383 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:03.304356    5678 patch_prober.go:29] interesting pod/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cqvp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Oct 30 22:11:03.304515 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:03.304425    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cqvp9" podUID="4e913a61-bea1-4281-9d5d-cdab5506a864" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Oct 30 22:11:03.492919 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:11:03.492883    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8061c232_3802_4de3_b53c_52238ba2bc75.slice/crio-c7e88b11fd84835dc82e05d8cc522446cca366eac2c66bb55e34dd8b70e3d91d WatchSource:0}: Error finding container c7e88b11fd84835dc82e05d8cc522446cca366eac2c66bb55e34dd8b70e3d91d: Status 404 returned error can't find the container with id c7e88b11fd84835dc82e05d8cc522446cca366eac2c66bb55e34dd8b70e3d91d
Oct 30 22:11:03.496206 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:03.496178    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-2d6zs"]
Oct 30 22:11:03.496868 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:03.496836    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:11:03.497014 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:03.496907    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:11:03.497014 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:03.496952    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:11:03.888980 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:03.888934    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-2d6zs" event={"ID":"8061c232-3802-4de3-b53c-52238ba2bc75","Type":"ContainerStarted","Data":"04d94a695fde10287a9a81f3853795998ed2fc7f424bfdaff7a792c7d466a83e"}
Oct 30 22:11:03.889156 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:03.889036    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-2d6zs" event={"ID":"8061c232-3802-4de3-b53c-52238ba2bc75","Type":"ContainerStarted","Data":"c7e88b11fd84835dc82e05d8cc522446cca366eac2c66bb55e34dd8b70e3d91d"}
Oct 30 22:11:04.008479 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:04.008443    5678 patch_prober.go:29] interesting pod/e2e-worker-w2-bc714d-openshift-default-67558b95cf-l8b5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Oct 30 22:11:04.008961 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:04.008505    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-l8b5h" podUID="09fbaa51-372b-406e-a478-3a7d52a8a618" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Oct 30 22:11:04.050425 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:04.042922    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-2d6zs" podStartSLOduration=2.04290231 podStartE2EDuration="2.04290231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:11:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:11:04.020508413 +0000 UTC m=+1241.624815971" watchObservedRunningTime="2026-10-30 22:11:04.04290231 +0000 UTC m=+1241.647209864"
Oct 30 22:11:04.050425 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:04.047742    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-2d6zs"
Oct 30 22:11:04.068424 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:04.066562    5678 patch_prober.go:29] interesting pod/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-2d6zs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Oct 30 22:11:04.068424 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:04.066624    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-2d6zs" podUID="8061c232-3802-4de3-b53c-52238ba2bc75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Oct 30 22:11:04.302170 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:04.302087    5678 patch_prober.go:29] interesting pod/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cqvp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Oct 30 22:11:04.302170 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:04.302148    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cqvp9" podUID="4e913a61-bea1-4281-9d5d-cdab5506a864" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Oct 30 22:11:05.008280 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:05.008247    5678 patch_prober.go:29] interesting pod/e2e-worker-w2-bc714d-openshift-default-67558b95cf-l8b5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Oct 30 22:11:05.008572 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:05.008540    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-l8b5h" podUID="09fbaa51-372b-406e-a478-3a7d52a8a618" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Oct 30 22:11:05.092426 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:05.073842    5678 patch_prober.go:29] interesting pod/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-2d6zs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Oct 30 22:11:05.092426 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:05.073907    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-2d6zs" podUID="8061c232-3802-4de3-b53c-52238ba2bc75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Oct 30 22:11:05.310482 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:05.309684    5678 patch_prober.go:29] interesting pod/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cqvp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Oct 30 22:11:05.310482 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:05.309746    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cqvp9" podUID="4e913a61-bea1-4281-9d5d-cdab5506a864" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Oct 30 22:11:06.014427 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:06.011026    5678 patch_prober.go:29] interesting pod/e2e-worker-w2-bc714d-openshift-default-67558b95cf-l8b5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Oct 30 22:11:06.014427 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:06.011087    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-l8b5h" podUID="09fbaa51-372b-406e-a478-3a7d52a8a618" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Oct 30 22:11:06.054425 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:06.054223    5678 patch_prober.go:29] interesting pod/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-2d6zs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Oct 30 22:11:06.054425 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:06.054285    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-2d6zs" podUID="8061c232-3802-4de3-b53c-52238ba2bc75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Oct 30 22:11:06.303007 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:06.302913    5678 patch_prober.go:29] interesting pod/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cqvp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Oct 30 22:11:06.303277 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:06.303247    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cqvp9" podUID="4e913a61-bea1-4281-9d5d-cdab5506a864" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Oct 30 22:11:07.008664 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:07.008626    5678 patch_prober.go:29] interesting pod/e2e-worker-w2-bc714d-openshift-default-67558b95cf-l8b5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Oct 30 22:11:07.008836 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:07.008758    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-l8b5h" podUID="09fbaa51-372b-406e-a478-3a7d52a8a618" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Oct 30 22:11:07.069426 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:07.050492    5678 patch_prober.go:29] interesting pod/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-2d6zs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Oct 30 22:11:07.069426 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:07.050554    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-2d6zs" podUID="8061c232-3802-4de3-b53c-52238ba2bc75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Oct 30 22:11:07.302932 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:07.302842    5678 patch_prober.go:29] interesting pod/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cqvp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Oct 30 22:11:07.302932 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:07.302894    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cqvp9" podUID="4e913a61-bea1-4281-9d5d-cdab5506a864" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Oct 30 22:11:08.007910 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:08.007869    5678 patch_prober.go:29] interesting pod/e2e-worker-w2-bc714d-openshift-default-67558b95cf-l8b5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Oct 30 22:11:08.008091 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:08.007933    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-l8b5h" podUID="09fbaa51-372b-406e-a478-3a7d52a8a618" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Oct 30 22:11:08.052424 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:08.051803    5678 patch_prober.go:29] interesting pod/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-2d6zs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Oct 30 22:11:08.052424 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:08.051865    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-2d6zs" podUID="8061c232-3802-4de3-b53c-52238ba2bc75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Oct 30 22:11:08.302465 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:08.302368    5678 patch_prober.go:29] interesting pod/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cqvp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Oct 30 22:11:08.302924 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:08.302901    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cqvp9" podUID="4e913a61-bea1-4281-9d5d-cdab5506a864" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Oct 30 22:11:09.007800 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:09.007760    5678 patch_prober.go:29] interesting pod/e2e-worker-w2-bc714d-openshift-default-67558b95cf-l8b5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Oct 30 22:11:09.008040 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:09.007864    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-l8b5h" podUID="09fbaa51-372b-406e-a478-3a7d52a8a618" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Oct 30 22:11:09.048427 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:09.047720    5678 patch_prober.go:29] interesting pod/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-2d6zs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Oct 30 22:11:09.048427 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:09.047779    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-2d6zs" podUID="8061c232-3802-4de3-b53c-52238ba2bc75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Oct 30 22:11:09.454426 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:09.453732    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cqvp9"
Oct 30 22:11:09.454989 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:09.454969    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cqvp9"
Oct 30 22:11:09.457129 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:09.457105    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cqvp9"
Oct 30 22:11:10.048703 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:10.048668    5678 patch_prober.go:29] interesting pod/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-2d6zs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Oct 30 22:11:10.048937 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:10.048733    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-2d6zs" podUID="8061c232-3802-4de3-b53c-52238ba2bc75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Oct 30 22:11:10.049497 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:10.049466    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-fb0b93/distinct-2-w6-fb0b93-kserve-7d65bb4698-m6tkh"]
Oct 30 22:11:10.054495 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:10.054471    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-fb0b93/distinct-2-w6-fb0b93-kserve-7d65bb4698-m6tkh"
Oct 30 22:11:10.157886 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:10.157845    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-fb0b93\"/\"distinct-2-w6-fb0b93-kserve-self-signed-certs\""
Oct 30 22:11:10.171268 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:10.171170    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-fb0b93/distinct-2-w6-fb0b93-kserve-7d65bb4698-m6tkh"]
Oct 30 22:11:10.198627 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:10.198591    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6fd276ef-522b-4bb4-b101-0a9c3fe9ecd3-dshm\") pod \"distinct-2-w6-fb0b93-kserve-7d65bb4698-m6tkh\" (UID: \"6fd276ef-522b-4bb4-b101-0a9c3fe9ecd3\") " pod="e2e-models-e2e-worker-w6-fb0b93/distinct-2-w6-fb0b93-kserve-7d65bb4698-m6tkh"
Oct 30 22:11:10.201561 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:10.198915    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6fd276ef-522b-4bb4-b101-0a9c3fe9ecd3-home\") pod \"distinct-2-w6-fb0b93-kserve-7d65bb4698-m6tkh\" (UID: \"6fd276ef-522b-4bb4-b101-0a9c3fe9ecd3\") " pod="e2e-models-e2e-worker-w6-fb0b93/distinct-2-w6-fb0b93-kserve-7d65bb4698-m6tkh"
Oct 30 22:11:10.201561 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:10.198990    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6fd276ef-522b-4bb4-b101-0a9c3fe9ecd3-tmp-dir\") pod \"distinct-2-w6-fb0b93-kserve-7d65bb4698-m6tkh\" (UID: \"6fd276ef-522b-4bb4-b101-0a9c3fe9ecd3\") " pod="e2e-models-e2e-worker-w6-fb0b93/distinct-2-w6-fb0b93-kserve-7d65bb4698-m6tkh"
Oct 30 22:11:10.201561 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:10.199040    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6fd276ef-522b-4bb4-b101-0a9c3fe9ecd3-tls-certs\") pod \"distinct-2-w6-fb0b93-kserve-7d65bb4698-m6tkh\" (UID: \"6fd276ef-522b-4bb4-b101-0a9c3fe9ecd3\") " pod="e2e-models-e2e-worker-w6-fb0b93/distinct-2-w6-fb0b93-kserve-7d65bb4698-m6tkh"
Oct 30 22:11:10.201561 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:10.199131    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6fd276ef-522b-4bb4-b101-0a9c3fe9ecd3-model-cache\") pod \"distinct-2-w6-fb0b93-kserve-7d65bb4698-m6tkh\" (UID: \"6fd276ef-522b-4bb4-b101-0a9c3fe9ecd3\") " pod="e2e-models-e2e-worker-w6-fb0b93/distinct-2-w6-fb0b93-kserve-7d65bb4698-m6tkh"
Oct 30 22:11:10.201561 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:10.199162    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qz6xn\" (UniqueName: \"kubernetes.io/projected/6fd276ef-522b-4bb4-b101-0a9c3fe9ecd3-kube-api-access-qz6xn\") pod \"distinct-2-w6-fb0b93-kserve-7d65bb4698-m6tkh\" (UID: \"6fd276ef-522b-4bb4-b101-0a9c3fe9ecd3\") " pod="e2e-models-e2e-worker-w6-fb0b93/distinct-2-w6-fb0b93-kserve-7d65bb4698-m6tkh"
Oct 30 22:11:10.201561 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:10.200205    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-l8b5h"
Oct 30 22:11:10.201561 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:10.201093    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-l8b5h"
Oct 30 22:11:10.202186 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:10.202163    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-l8b5h"
Oct 30 22:11:10.300237 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:10.300138    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6fd276ef-522b-4bb4-b101-0a9c3fe9ecd3-tmp-dir\") pod \"distinct-2-w6-fb0b93-kserve-7d65bb4698-m6tkh\" (UID: \"6fd276ef-522b-4bb4-b101-0a9c3fe9ecd3\") " pod="e2e-models-e2e-worker-w6-fb0b93/distinct-2-w6-fb0b93-kserve-7d65bb4698-m6tkh"
Oct 30 22:11:10.300237 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:10.300173    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6fd276ef-522b-4bb4-b101-0a9c3fe9ecd3-tls-certs\") pod \"distinct-2-w6-fb0b93-kserve-7d65bb4698-m6tkh\" (UID: \"6fd276ef-522b-4bb4-b101-0a9c3fe9ecd3\") " pod="e2e-models-e2e-worker-w6-fb0b93/distinct-2-w6-fb0b93-kserve-7d65bb4698-m6tkh"
Oct 30 22:11:10.300551 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:10.300241    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6fd276ef-522b-4bb4-b101-0a9c3fe9ecd3-model-cache\") pod \"distinct-2-w6-fb0b93-kserve-7d65bb4698-m6tkh\" (UID: \"6fd276ef-522b-4bb4-b101-0a9c3fe9ecd3\") " pod="e2e-models-e2e-worker-w6-fb0b93/distinct-2-w6-fb0b93-kserve-7d65bb4698-m6tkh"
Oct 30 22:11:10.300551 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:10.300261    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qz6xn\" (UniqueName: \"kubernetes.io/projected/6fd276ef-522b-4bb4-b101-0a9c3fe9ecd3-kube-api-access-qz6xn\") pod \"distinct-2-w6-fb0b93-kserve-7d65bb4698-m6tkh\" (UID: \"6fd276ef-522b-4bb4-b101-0a9c3fe9ecd3\") " pod="e2e-models-e2e-worker-w6-fb0b93/distinct-2-w6-fb0b93-kserve-7d65bb4698-m6tkh"
Oct 30 22:11:10.300551 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:10.300324    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6fd276ef-522b-4bb4-b101-0a9c3fe9ecd3-dshm\") pod \"distinct-2-w6-fb0b93-kserve-7d65bb4698-m6tkh\" (UID: \"6fd276ef-522b-4bb4-b101-0a9c3fe9ecd3\") " pod="e2e-models-e2e-worker-w6-fb0b93/distinct-2-w6-fb0b93-kserve-7d65bb4698-m6tkh"
Oct 30 22:11:10.300739 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:10.300604    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6fd276ef-522b-4bb4-b101-0a9c3fe9ecd3-tmp-dir\") pod \"distinct-2-w6-fb0b93-kserve-7d65bb4698-m6tkh\" (UID: \"6fd276ef-522b-4bb4-b101-0a9c3fe9ecd3\") " pod="e2e-models-e2e-worker-w6-fb0b93/distinct-2-w6-fb0b93-kserve-7d65bb4698-m6tkh"
Oct 30 22:11:10.300805 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:10.300761    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6fd276ef-522b-4bb4-b101-0a9c3fe9ecd3-home\") pod \"distinct-2-w6-fb0b93-kserve-7d65bb4698-m6tkh\" (UID: \"6fd276ef-522b-4bb4-b101-0a9c3fe9ecd3\") " pod="e2e-models-e2e-worker-w6-fb0b93/distinct-2-w6-fb0b93-kserve-7d65bb4698-m6tkh"
Oct 30 22:11:10.300805 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:10.300780    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6fd276ef-522b-4bb4-b101-0a9c3fe9ecd3-model-cache\") pod \"distinct-2-w6-fb0b93-kserve-7d65bb4698-m6tkh\" (UID: \"6fd276ef-522b-4bb4-b101-0a9c3fe9ecd3\") " pod="e2e-models-e2e-worker-w6-fb0b93/distinct-2-w6-fb0b93-kserve-7d65bb4698-m6tkh"
Oct 30 22:11:10.301214 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:10.301185    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6fd276ef-522b-4bb4-b101-0a9c3fe9ecd3-home\") pod \"distinct-2-w6-fb0b93-kserve-7d65bb4698-m6tkh\" (UID: \"6fd276ef-522b-4bb4-b101-0a9c3fe9ecd3\") " pod="e2e-models-e2e-worker-w6-fb0b93/distinct-2-w6-fb0b93-kserve-7d65bb4698-m6tkh"
Oct 30 22:11:10.303061 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:10.303014    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6fd276ef-522b-4bb4-b101-0a9c3fe9ecd3-dshm\") pod \"distinct-2-w6-fb0b93-kserve-7d65bb4698-m6tkh\" (UID: \"6fd276ef-522b-4bb4-b101-0a9c3fe9ecd3\") " pod="e2e-models-e2e-worker-w6-fb0b93/distinct-2-w6-fb0b93-kserve-7d65bb4698-m6tkh"
Oct 30 22:11:10.305994 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:10.305969    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6fd276ef-522b-4bb4-b101-0a9c3fe9ecd3-tls-certs\") pod \"distinct-2-w6-fb0b93-kserve-7d65bb4698-m6tkh\" (UID: \"6fd276ef-522b-4bb4-b101-0a9c3fe9ecd3\") " pod="e2e-models-e2e-worker-w6-fb0b93/distinct-2-w6-fb0b93-kserve-7d65bb4698-m6tkh"
Oct 30 22:11:10.394008 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:10.393969    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qz6xn\" (UniqueName: \"kubernetes.io/projected/6fd276ef-522b-4bb4-b101-0a9c3fe9ecd3-kube-api-access-qz6xn\") pod \"distinct-2-w6-fb0b93-kserve-7d65bb4698-m6tkh\" (UID: \"6fd276ef-522b-4bb4-b101-0a9c3fe9ecd3\") " pod="e2e-models-e2e-worker-w6-fb0b93/distinct-2-w6-fb0b93-kserve-7d65bb4698-m6tkh"
Oct 30 22:11:10.629296 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:11:10.629253    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 22:11:10.676845 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:10.676812    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-fb0b93/distinct-2-w6-fb0b93-kserve-7d65bb4698-m6tkh"
Oct 30 22:11:11.066752 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:11.066727    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-fb0b93/distinct-2-w6-fb0b93-kserve-7d65bb4698-m6tkh"]
Oct 30 22:11:11.068751 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:11:11.068727    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6fd276ef_522b_4bb4_b101_0a9c3fe9ecd3.slice/crio-61f8438948d1cfc08745c4cd6defc3cb0c5c78d949b2c300a244ece461463dde WatchSource:0}: Error finding container 61f8438948d1cfc08745c4cd6defc3cb0c5c78d949b2c300a244ece461463dde: Status 404 returned error can't find the container with id 61f8438948d1cfc08745c4cd6defc3cb0c5c78d949b2c300a244ece461463dde
Oct 30 22:11:11.222160 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:11.222128    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-2d6zs"
Oct 30 22:11:11.223040 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:11.223015    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-2d6zs"
Oct 30 22:11:11.223975 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:11.223953    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-2d6zs"
Oct 30 22:11:12.038325 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:12.038286    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-fb0b93/distinct-2-w6-fb0b93-kserve-7d65bb4698-m6tkh" event={"ID":"6fd276ef-522b-4bb4-b101-0a9c3fe9ecd3","Type":"ContainerStarted","Data":"8997f6d274894fb2ebc9077cd8197d93f8c588510a2364ea8dc174114d982ed5"}
Oct 30 22:11:12.038758 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:12.038338    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-fb0b93/distinct-2-w6-fb0b93-kserve-7d65bb4698-m6tkh" event={"ID":"6fd276ef-522b-4bb4-b101-0a9c3fe9ecd3","Type":"ContainerStarted","Data":"61f8438948d1cfc08745c4cd6defc3cb0c5c78d949b2c300a244ece461463dde"}
Oct 30 22:11:12.152898 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:12.152847    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-fb0b93/distinct-2-w6-fb0b93-kserve-7d65bb4698-m6tkh" podStartSLOduration=2.152834707 podStartE2EDuration="2.152834707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:11:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:11:12.150287442 +0000 UTC m=+1249.754594995" watchObservedRunningTime="2026-10-30 22:11:12.152834707 +0000 UTC m=+1249.757142294"
Oct 30 22:11:12.678010 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:12.677983    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-fb0b93/distinct-2-w6-fb0b93-kserve-7d65bb4698-m6tkh"
Oct 30 22:11:20.677465 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:20.677428    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-fb0b93/distinct-2-w6-fb0b93-kserve-7d65bb4698-m6tkh"
Oct 30 22:11:20.689820 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:20.689792    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-fb0b93/distinct-2-w6-fb0b93-kserve-7d65bb4698-m6tkh"
Oct 30 22:11:21.110742 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:21.110648    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-fb0b93/distinct-2-w6-fb0b93-kserve-7d65bb4698-m6tkh"
Oct 30 22:11:40.574028 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:11:40.573945    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 22:11:49.624252 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:49.624217    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbca0da1-tg22z"]
Oct 30 22:11:49.628789 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:49.628765    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbca0da1-tg22z"
Oct 30 22:11:49.679681 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:49.679659    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:11:49.726384 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:49.726356    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5rvcj\""
Oct 30 22:11:49.726873 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:49.726851    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttzl6\" (UniqueName: \"kubernetes.io/projected/9147be34-042d-49b5-9f75-2f2440a6dae9-kube-api-access-ttzl6\") pod \"maas-api-revoke-sub-dbca0da1-tg22z\" (UID: \"9147be34-042d-49b5-9f75-2f2440a6dae9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbca0da1-tg22z"
Oct 30 22:11:49.726926 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:49.726889    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9147be34-042d-49b5-9f75-2f2440a6dae9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dbca0da1-tg22z\" (UID: \"9147be34-042d-49b5-9f75-2f2440a6dae9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbca0da1-tg22z"
Oct 30 22:11:49.727127 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:49.727102    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9147be34-042d-49b5-9f75-2f2440a6dae9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dbca0da1-tg22z\" (UID: \"9147be34-042d-49b5-9f75-2f2440a6dae9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbca0da1-tg22z"
Oct 30 22:11:49.730615 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:49.730599    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbca0da1-tg22z"]
Oct 30 22:11:49.828281 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:49.828251    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttzl6\" (UniqueName: \"kubernetes.io/projected/9147be34-042d-49b5-9f75-2f2440a6dae9-kube-api-access-ttzl6\") pod \"maas-api-revoke-sub-dbca0da1-tg22z\" (UID: \"9147be34-042d-49b5-9f75-2f2440a6dae9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbca0da1-tg22z"
Oct 30 22:11:49.828448 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:49.828295    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9147be34-042d-49b5-9f75-2f2440a6dae9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dbca0da1-tg22z\" (UID: \"9147be34-042d-49b5-9f75-2f2440a6dae9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbca0da1-tg22z"
Oct 30 22:11:49.828497 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:49.828467    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9147be34-042d-49b5-9f75-2f2440a6dae9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dbca0da1-tg22z\" (UID: \"9147be34-042d-49b5-9f75-2f2440a6dae9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbca0da1-tg22z"
Oct 30 22:11:49.829214 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:49.829194    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9147be34-042d-49b5-9f75-2f2440a6dae9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dbca0da1-tg22z\" (UID: \"9147be34-042d-49b5-9f75-2f2440a6dae9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbca0da1-tg22z"
Oct 30 22:11:49.830717 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:49.830692    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9147be34-042d-49b5-9f75-2f2440a6dae9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dbca0da1-tg22z\" (UID: \"9147be34-042d-49b5-9f75-2f2440a6dae9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbca0da1-tg22z"
Oct 30 22:11:49.887050 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:49.886990    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttzl6\" (UniqueName: \"kubernetes.io/projected/9147be34-042d-49b5-9f75-2f2440a6dae9-kube-api-access-ttzl6\") pod \"maas-api-revoke-sub-dbca0da1-tg22z\" (UID: \"9147be34-042d-49b5-9f75-2f2440a6dae9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbca0da1-tg22z"
Oct 30 22:11:49.942558 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:49.942537    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbca0da1-tg22z"
Oct 30 22:11:50.323611 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:11:50.323578    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9147be34_042d_49b5_9f75_2f2440a6dae9.slice/crio-cc09cc112301d945e922220fbd5564981a99864f97b07439b9ea57376c29efd5 WatchSource:0}: Error finding container cc09cc112301d945e922220fbd5564981a99864f97b07439b9ea57376c29efd5: Status 404 returned error can't find the container with id cc09cc112301d945e922220fbd5564981a99864f97b07439b9ea57376c29efd5
Oct 30 22:11:50.324053 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:50.324020    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbca0da1-tg22z"]
Oct 30 22:11:50.978120 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:50.978033    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-a00d9b/distinct-2-w1-a00d9b-kserve-68fd95957c-dmgnb"]
Oct 30 22:11:50.982215 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:50.982193    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a00d9b/distinct-2-w1-a00d9b-kserve-68fd95957c-dmgnb"
Oct 30 22:11:51.044203 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:51.044166    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23a81fda-f7e3-4f91-8d96-9f19c7de6e09-home\") pod \"distinct-2-w1-a00d9b-kserve-68fd95957c-dmgnb\" (UID: \"23a81fda-f7e3-4f91-8d96-9f19c7de6e09\") " pod="e2e-models-e2e-worker-w1-a00d9b/distinct-2-w1-a00d9b-kserve-68fd95957c-dmgnb"
Oct 30 22:11:51.044384 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:51.044212    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23a81fda-f7e3-4f91-8d96-9f19c7de6e09-tls-certs\") pod \"distinct-2-w1-a00d9b-kserve-68fd95957c-dmgnb\" (UID: \"23a81fda-f7e3-4f91-8d96-9f19c7de6e09\") " pod="e2e-models-e2e-worker-w1-a00d9b/distinct-2-w1-a00d9b-kserve-68fd95957c-dmgnb"
Oct 30 22:11:51.044384 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:51.044269    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gh2zc\" (UniqueName: \"kubernetes.io/projected/23a81fda-f7e3-4f91-8d96-9f19c7de6e09-kube-api-access-gh2zc\") pod \"distinct-2-w1-a00d9b-kserve-68fd95957c-dmgnb\" (UID: \"23a81fda-f7e3-4f91-8d96-9f19c7de6e09\") " pod="e2e-models-e2e-worker-w1-a00d9b/distinct-2-w1-a00d9b-kserve-68fd95957c-dmgnb"
Oct 30 22:11:51.044384 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:51.044381    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23a81fda-f7e3-4f91-8d96-9f19c7de6e09-dshm\") pod \"distinct-2-w1-a00d9b-kserve-68fd95957c-dmgnb\" (UID: \"23a81fda-f7e3-4f91-8d96-9f19c7de6e09\") " pod="e2e-models-e2e-worker-w1-a00d9b/distinct-2-w1-a00d9b-kserve-68fd95957c-dmgnb"
Oct 30 22:11:51.044617 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:51.044501    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23a81fda-f7e3-4f91-8d96-9f19c7de6e09-tmp-dir\") pod \"distinct-2-w1-a00d9b-kserve-68fd95957c-dmgnb\" (UID: \"23a81fda-f7e3-4f91-8d96-9f19c7de6e09\") " pod="e2e-models-e2e-worker-w1-a00d9b/distinct-2-w1-a00d9b-kserve-68fd95957c-dmgnb"
Oct 30 22:11:51.044617 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:51.044568    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23a81fda-f7e3-4f91-8d96-9f19c7de6e09-model-cache\") pod \"distinct-2-w1-a00d9b-kserve-68fd95957c-dmgnb\" (UID: \"23a81fda-f7e3-4f91-8d96-9f19c7de6e09\") " pod="e2e-models-e2e-worker-w1-a00d9b/distinct-2-w1-a00d9b-kserve-68fd95957c-dmgnb"
Oct 30 22:11:51.081482 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:51.081451    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-a00d9b\"/\"distinct-2-w1-a00d9b-kserve-self-signed-certs\""
Oct 30 22:11:51.081656 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:51.081495    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-a00d9b\"/\"openshift-service-ca.crt\""
Oct 30 22:11:51.081656 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:51.081502    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-a00d9b\"/\"kube-root-ca.crt\""
Oct 30 22:11:51.081656 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:51.081500    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-a00d9b\"/\"default-dockercfg-ttmlk\""
Oct 30 22:11:51.083928 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:51.083908    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-a00d9b/distinct-2-w1-a00d9b-kserve-68fd95957c-dmgnb"]
Oct 30 22:11:51.145671 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:51.145635    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23a81fda-f7e3-4f91-8d96-9f19c7de6e09-tmp-dir\") pod \"distinct-2-w1-a00d9b-kserve-68fd95957c-dmgnb\" (UID: \"23a81fda-f7e3-4f91-8d96-9f19c7de6e09\") " pod="e2e-models-e2e-worker-w1-a00d9b/distinct-2-w1-a00d9b-kserve-68fd95957c-dmgnb"
Oct 30 22:11:51.145944 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:51.145710    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23a81fda-f7e3-4f91-8d96-9f19c7de6e09-model-cache\") pod \"distinct-2-w1-a00d9b-kserve-68fd95957c-dmgnb\" (UID: \"23a81fda-f7e3-4f91-8d96-9f19c7de6e09\") " pod="e2e-models-e2e-worker-w1-a00d9b/distinct-2-w1-a00d9b-kserve-68fd95957c-dmgnb"
Oct 30 22:11:51.145944 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:51.145809    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23a81fda-f7e3-4f91-8d96-9f19c7de6e09-home\") pod \"distinct-2-w1-a00d9b-kserve-68fd95957c-dmgnb\" (UID: \"23a81fda-f7e3-4f91-8d96-9f19c7de6e09\") " pod="e2e-models-e2e-worker-w1-a00d9b/distinct-2-w1-a00d9b-kserve-68fd95957c-dmgnb"
Oct 30 22:11:51.145944 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:51.145825    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23a81fda-f7e3-4f91-8d96-9f19c7de6e09-tls-certs\") pod \"distinct-2-w1-a00d9b-kserve-68fd95957c-dmgnb\" (UID: \"23a81fda-f7e3-4f91-8d96-9f19c7de6e09\") " pod="e2e-models-e2e-worker-w1-a00d9b/distinct-2-w1-a00d9b-kserve-68fd95957c-dmgnb"
Oct 30 22:11:51.145944 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:51.145842    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gh2zc\" (UniqueName: \"kubernetes.io/projected/23a81fda-f7e3-4f91-8d96-9f19c7de6e09-kube-api-access-gh2zc\") pod \"distinct-2-w1-a00d9b-kserve-68fd95957c-dmgnb\" (UID: \"23a81fda-f7e3-4f91-8d96-9f19c7de6e09\") " pod="e2e-models-e2e-worker-w1-a00d9b/distinct-2-w1-a00d9b-kserve-68fd95957c-dmgnb"
Oct 30 22:11:51.147522 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:51.146158    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23a81fda-f7e3-4f91-8d96-9f19c7de6e09-tmp-dir\") pod \"distinct-2-w1-a00d9b-kserve-68fd95957c-dmgnb\" (UID: \"23a81fda-f7e3-4f91-8d96-9f19c7de6e09\") " pod="e2e-models-e2e-worker-w1-a00d9b/distinct-2-w1-a00d9b-kserve-68fd95957c-dmgnb"
Oct 30 22:11:51.147522 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:51.146212    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23a81fda-f7e3-4f91-8d96-9f19c7de6e09-model-cache\") pod \"distinct-2-w1-a00d9b-kserve-68fd95957c-dmgnb\" (UID: \"23a81fda-f7e3-4f91-8d96-9f19c7de6e09\") " pod="e2e-models-e2e-worker-w1-a00d9b/distinct-2-w1-a00d9b-kserve-68fd95957c-dmgnb"
Oct 30 22:11:51.147522 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:51.146229    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23a81fda-f7e3-4f91-8d96-9f19c7de6e09-home\") pod \"distinct-2-w1-a00d9b-kserve-68fd95957c-dmgnb\" (UID: \"23a81fda-f7e3-4f91-8d96-9f19c7de6e09\") " pod="e2e-models-e2e-worker-w1-a00d9b/distinct-2-w1-a00d9b-kserve-68fd95957c-dmgnb"
Oct 30 22:11:51.147522 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:51.147114    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23a81fda-f7e3-4f91-8d96-9f19c7de6e09-dshm\") pod \"distinct-2-w1-a00d9b-kserve-68fd95957c-dmgnb\" (UID: \"23a81fda-f7e3-4f91-8d96-9f19c7de6e09\") " pod="e2e-models-e2e-worker-w1-a00d9b/distinct-2-w1-a00d9b-kserve-68fd95957c-dmgnb"
Oct 30 22:11:51.149568 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:51.149545    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23a81fda-f7e3-4f91-8d96-9f19c7de6e09-tls-certs\") pod \"distinct-2-w1-a00d9b-kserve-68fd95957c-dmgnb\" (UID: \"23a81fda-f7e3-4f91-8d96-9f19c7de6e09\") " pod="e2e-models-e2e-worker-w1-a00d9b/distinct-2-w1-a00d9b-kserve-68fd95957c-dmgnb"
Oct 30 22:11:51.149678 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:51.149619    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23a81fda-f7e3-4f91-8d96-9f19c7de6e09-dshm\") pod \"distinct-2-w1-a00d9b-kserve-68fd95957c-dmgnb\" (UID: \"23a81fda-f7e3-4f91-8d96-9f19c7de6e09\") " pod="e2e-models-e2e-worker-w1-a00d9b/distinct-2-w1-a00d9b-kserve-68fd95957c-dmgnb"
Oct 30 22:11:51.205932 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:51.205869    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gh2zc\" (UniqueName: \"kubernetes.io/projected/23a81fda-f7e3-4f91-8d96-9f19c7de6e09-kube-api-access-gh2zc\") pod \"distinct-2-w1-a00d9b-kserve-68fd95957c-dmgnb\" (UID: \"23a81fda-f7e3-4f91-8d96-9f19c7de6e09\") " pod="e2e-models-e2e-worker-w1-a00d9b/distinct-2-w1-a00d9b-kserve-68fd95957c-dmgnb"
Oct 30 22:11:51.296118 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:51.296085    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a00d9b/distinct-2-w1-a00d9b-kserve-68fd95957c-dmgnb"
Oct 30 22:11:51.357770 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:51.357728    5678 generic.go:356] "Generic (PLEG): container finished" podID="9147be34-042d-49b5-9f75-2f2440a6dae9" containerID="36b76e5664acf6be7359c5c8c453de6815ecb9e8ad2f45d70f7407bfdac8cecc" exitCode=0
Oct 30 22:11:51.357983 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:51.357780    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbca0da1-tg22z" event={"ID":"9147be34-042d-49b5-9f75-2f2440a6dae9","Type":"ContainerDied","Data":"36b76e5664acf6be7359c5c8c453de6815ecb9e8ad2f45d70f7407bfdac8cecc"}
Oct 30 22:11:51.357983 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:51.357831    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbca0da1-tg22z" event={"ID":"9147be34-042d-49b5-9f75-2f2440a6dae9","Type":"ContainerStarted","Data":"cc09cc112301d945e922220fbd5564981a99864f97b07439b9ea57376c29efd5"}
Oct 30 22:11:51.712625 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:51.712590    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-a00d9b/distinct-2-w1-a00d9b-kserve-68fd95957c-dmgnb"]
Oct 30 22:11:51.712778 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:11:51.712632    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod23a81fda_f7e3_4f91_8d96_9f19c7de6e09.slice/crio-301b15bf374e098a338e2f8494763acf3d54740ed0066a7f3bf425fd6ba866f5 WatchSource:0}: Error finding container 301b15bf374e098a338e2f8494763acf3d54740ed0066a7f3bf425fd6ba866f5: Status 404 returned error can't find the container with id 301b15bf374e098a338e2f8494763acf3d54740ed0066a7f3bf425fd6ba866f5
Oct 30 22:11:52.366584 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:52.366546    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a00d9b/distinct-2-w1-a00d9b-kserve-68fd95957c-dmgnb" event={"ID":"23a81fda-f7e3-4f91-8d96-9f19c7de6e09","Type":"ContainerStarted","Data":"11b87afbda21db67b69a33372ded4712e8f6fc805e169106936e32e8f2070d2a"}
Oct 30 22:11:52.366985 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:52.366598    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a00d9b/distinct-2-w1-a00d9b-kserve-68fd95957c-dmgnb" event={"ID":"23a81fda-f7e3-4f91-8d96-9f19c7de6e09","Type":"ContainerStarted","Data":"301b15bf374e098a338e2f8494763acf3d54740ed0066a7f3bf425fd6ba866f5"}
Oct 30 22:11:52.479797 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:52.479733    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-a00d9b/distinct-2-w1-a00d9b-kserve-68fd95957c-dmgnb" podStartSLOduration=2.479720134 podStartE2EDuration="2.479720134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:11:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:11:52.478753088 +0000 UTC m=+1290.083060641" watchObservedRunningTime="2026-10-30 22:11:52.479720134 +0000 UTC m=+1290.084027832"
Oct 30 22:11:52.533929 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:52.533906    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbca0da1-tg22z"
Oct 30 22:11:52.571585 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:52.571559    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9147be34-042d-49b5-9f75-2f2440a6dae9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9147be34-042d-49b5-9f75-2f2440a6dae9-maas-api-xks-ca\") pod \"9147be34-042d-49b5-9f75-2f2440a6dae9\" (UID: \"9147be34-042d-49b5-9f75-2f2440a6dae9\") "
Oct 30 22:11:52.571729 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:52.571631    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9147be34-042d-49b5-9f75-2f2440a6dae9-kube-api-access-ttzl6\" (UniqueName: \"kubernetes.io/projected/9147be34-042d-49b5-9f75-2f2440a6dae9-kube-api-access-ttzl6\") pod \"9147be34-042d-49b5-9f75-2f2440a6dae9\" (UID: \"9147be34-042d-49b5-9f75-2f2440a6dae9\") "
Oct 30 22:11:52.571729 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:52.571655    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9147be34-042d-49b5-9f75-2f2440a6dae9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9147be34-042d-49b5-9f75-2f2440a6dae9-maas-api-openshift-ca\") pod \"9147be34-042d-49b5-9f75-2f2440a6dae9\" (UID: \"9147be34-042d-49b5-9f75-2f2440a6dae9\") "
Oct 30 22:11:52.572097 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:52.572073    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9147be34-042d-49b5-9f75-2f2440a6dae9-maas-api-openshift-ca" pod "9147be34-042d-49b5-9f75-2f2440a6dae9" (UID: "9147be34-042d-49b5-9f75-2f2440a6dae9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:11:52.572298 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:52.572282    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9147be34-042d-49b5-9f75-2f2440a6dae9-maas-api-openshift-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:52.573737 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:52.573715    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9147be34-042d-49b5-9f75-2f2440a6dae9-kube-api-access-ttzl6" pod "9147be34-042d-49b5-9f75-2f2440a6dae9" (UID: "9147be34-042d-49b5-9f75-2f2440a6dae9"). InnerVolumeSpecName "kube-api-access-ttzl6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:11:52.573823 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:52.573717    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9147be34-042d-49b5-9f75-2f2440a6dae9-maas-api-xks-ca" pod "9147be34-042d-49b5-9f75-2f2440a6dae9" (UID: "9147be34-042d-49b5-9f75-2f2440a6dae9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:11:52.673963 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:52.673877    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9147be34-042d-49b5-9f75-2f2440a6dae9-maas-api-xks-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:52.673963 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:52.673919    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttzl6\" (UniqueName: \"kubernetes.io/projected/9147be34-042d-49b5-9f75-2f2440a6dae9-kube-api-access-ttzl6\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:53.296687 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:53.296656    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a00d9b/distinct-2-w1-a00d9b-kserve-68fd95957c-dmgnb"
Oct 30 22:11:53.380294 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:53.380268    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbca0da1-tg22z"
Oct 30 22:11:53.380697 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:53.380302    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbca0da1-tg22z" event={"ID":"9147be34-042d-49b5-9f75-2f2440a6dae9","Type":"ContainerDied","Data":"cc09cc112301d945e922220fbd5564981a99864f97b07439b9ea57376c29efd5"}
Oct 30 22:11:53.380697 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:11:53.380348    5678 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cc09cc112301d945e922220fbd5564981a99864f97b07439b9ea57376c29efd5"
Oct 30 22:12:01.296889 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:01.296849    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-a00d9b/distinct-2-w1-a00d9b-kserve-68fd95957c-dmgnb"
Oct 30 22:12:01.309360 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:01.309335    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-a00d9b/distinct-2-w1-a00d9b-kserve-68fd95957c-dmgnb"
Oct 30 22:12:01.554486 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:01.553802    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a00d9b/distinct-2-w1-a00d9b-kserve-68fd95957c-dmgnb"
Oct 30 22:12:10.573184 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:12:10.573151    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 22:12:17.797786 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:17.797751    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-f2qb5"]
Oct 30 22:12:17.798522 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:17.798504    5678 cpu_manager.go:414] "containerMap: removing container" podUID="9147be34-042d-49b5-9f75-2f2440a6dae9" containerName="revoke-keys"
Oct 30 22:12:17.798585 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:17.798523    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9147be34-042d-49b5-9f75-2f2440a6dae9" containerName="revoke-keys"
Oct 30 22:12:17.798691 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:17.798680    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="9147be34-042d-49b5-9f75-2f2440a6dae9" containerName="revoke-keys"
Oct 30 22:12:17.802430 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:17.802391    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-f2qb5"
Oct 30 22:12:17.841897 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:17.841869    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5041edee-ad23-43bf-a84e-9bb6632c2c0b-workload-certs\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-f2qb5\" (UID: \"5041edee-ad23-43bf-a84e-9bb6632c2c0b\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-f2qb5"
Oct 30 22:12:17.842038 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:17.841915    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5041edee-ad23-43bf-a84e-9bb6632c2c0b-istio-envoy\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-f2qb5\" (UID: \"5041edee-ad23-43bf-a84e-9bb6632c2c0b\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-f2qb5"
Oct 30 22:12:17.842038 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:17.841956    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5041edee-ad23-43bf-a84e-9bb6632c2c0b-istio-token\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-f2qb5\" (UID: \"5041edee-ad23-43bf-a84e-9bb6632c2c0b\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-f2qb5"
Oct 30 22:12:17.842115 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:17.842085    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5041edee-ad23-43bf-a84e-9bb6632c2c0b-istio-data\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-f2qb5\" (UID: \"5041edee-ad23-43bf-a84e-9bb6632c2c0b\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-f2qb5"
Oct 30 22:12:17.842150 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:17.842126    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5041edee-ad23-43bf-a84e-9bb6632c2c0b-credential-socket\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-f2qb5\" (UID: \"5041edee-ad23-43bf-a84e-9bb6632c2c0b\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-f2qb5"
Oct 30 22:12:17.842184 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:17.842147    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5041edee-ad23-43bf-a84e-9bb6632c2c0b-workload-socket\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-f2qb5\" (UID: \"5041edee-ad23-43bf-a84e-9bb6632c2c0b\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-f2qb5"
Oct 30 22:12:17.842254 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:17.842240    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5041edee-ad23-43bf-a84e-9bb6632c2c0b-istio-podinfo\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-f2qb5\" (UID: \"5041edee-ad23-43bf-a84e-9bb6632c2c0b\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-f2qb5"
Oct 30 22:12:17.842306 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:17.842276    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftsb4\" (UniqueName: \"kubernetes.io/projected/5041edee-ad23-43bf-a84e-9bb6632c2c0b-kube-api-access-ftsb4\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-f2qb5\" (UID: \"5041edee-ad23-43bf-a84e-9bb6632c2c0b\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-f2qb5"
Oct 30 22:12:17.842466 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:17.842444    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5041edee-ad23-43bf-a84e-9bb6632c2c0b-istiod-ca-cert\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-f2qb5\" (UID: \"5041edee-ad23-43bf-a84e-9bb6632c2c0b\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-f2qb5"
Oct 30 22:12:17.899142 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:17.899102    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-f2qb5"]
Oct 30 22:12:17.917724 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:17.917696    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-m2jqj"]
Oct 30 22:12:17.922155 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:17.922133    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-m2jqj"
Oct 30 22:12:17.944184 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:17.944158    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5041edee-ad23-43bf-a84e-9bb6632c2c0b-workload-socket\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-f2qb5\" (UID: \"5041edee-ad23-43bf-a84e-9bb6632c2c0b\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-f2qb5"
Oct 30 22:12:17.944338 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:17.944194    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5041edee-ad23-43bf-a84e-9bb6632c2c0b-istio-podinfo\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-f2qb5\" (UID: \"5041edee-ad23-43bf-a84e-9bb6632c2c0b\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-f2qb5"
Oct 30 22:12:17.944338 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:17.944233    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftsb4\" (UniqueName: \"kubernetes.io/projected/5041edee-ad23-43bf-a84e-9bb6632c2c0b-kube-api-access-ftsb4\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-f2qb5\" (UID: \"5041edee-ad23-43bf-a84e-9bb6632c2c0b\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-f2qb5"
Oct 30 22:12:17.944502 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:17.944373    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5041edee-ad23-43bf-a84e-9bb6632c2c0b-istiod-ca-cert\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-f2qb5\" (UID: \"5041edee-ad23-43bf-a84e-9bb6632c2c0b\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-f2qb5"
Oct 30 22:12:17.944502 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:17.944490    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5041edee-ad23-43bf-a84e-9bb6632c2c0b-workload-certs\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-f2qb5\" (UID: \"5041edee-ad23-43bf-a84e-9bb6632c2c0b\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-f2qb5"
Oct 30 22:12:17.944684 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:17.944544    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5041edee-ad23-43bf-a84e-9bb6632c2c0b-istio-envoy\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-f2qb5\" (UID: \"5041edee-ad23-43bf-a84e-9bb6632c2c0b\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-f2qb5"
Oct 30 22:12:17.944684 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:17.944575    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5041edee-ad23-43bf-a84e-9bb6632c2c0b-istio-token\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-f2qb5\" (UID: \"5041edee-ad23-43bf-a84e-9bb6632c2c0b\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-f2qb5"
Oct 30 22:12:17.944684 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:17.944662    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5041edee-ad23-43bf-a84e-9bb6632c2c0b-istio-data\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-f2qb5\" (UID: \"5041edee-ad23-43bf-a84e-9bb6632c2c0b\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-f2qb5"
Oct 30 22:12:17.944847 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:17.944725    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5041edee-ad23-43bf-a84e-9bb6632c2c0b-credential-socket\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-f2qb5\" (UID: \"5041edee-ad23-43bf-a84e-9bb6632c2c0b\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-f2qb5"
Oct 30 22:12:17.944847 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:17.944805    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5041edee-ad23-43bf-a84e-9bb6632c2c0b-workload-certs\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-f2qb5\" (UID: \"5041edee-ad23-43bf-a84e-9bb6632c2c0b\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-f2qb5"
Oct 30 22:12:17.945135 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:17.945086    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5041edee-ad23-43bf-a84e-9bb6632c2c0b-istio-data\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-f2qb5\" (UID: \"5041edee-ad23-43bf-a84e-9bb6632c2c0b\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-f2qb5"
Oct 30 22:12:17.946362 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:17.946005    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5041edee-ad23-43bf-a84e-9bb6632c2c0b-istiod-ca-cert\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-f2qb5\" (UID: \"5041edee-ad23-43bf-a84e-9bb6632c2c0b\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-f2qb5"
Oct 30 22:12:17.946362 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:17.946216    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5041edee-ad23-43bf-a84e-9bb6632c2c0b-workload-socket\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-f2qb5\" (UID: \"5041edee-ad23-43bf-a84e-9bb6632c2c0b\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-f2qb5"
Oct 30 22:12:17.949396 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:17.949194    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5041edee-ad23-43bf-a84e-9bb6632c2c0b-istio-envoy\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-f2qb5\" (UID: \"5041edee-ad23-43bf-a84e-9bb6632c2c0b\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-f2qb5"
Oct 30 22:12:17.949396 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:17.949203    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5041edee-ad23-43bf-a84e-9bb6632c2c0b-credential-socket\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-f2qb5\" (UID: \"5041edee-ad23-43bf-a84e-9bb6632c2c0b\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-f2qb5"
Oct 30 22:12:17.949396 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:17.949351    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5041edee-ad23-43bf-a84e-9bb6632c2c0b-istio-podinfo\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-f2qb5\" (UID: \"5041edee-ad23-43bf-a84e-9bb6632c2c0b\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-f2qb5"
Oct 30 22:12:18.002729 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:18.002691    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-m2jqj"]
Oct 30 22:12:18.006640 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:18.006611    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5041edee-ad23-43bf-a84e-9bb6632c2c0b-istio-token\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-f2qb5\" (UID: \"5041edee-ad23-43bf-a84e-9bb6632c2c0b\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-f2qb5"
Oct 30 22:12:18.007027 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:18.007001    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftsb4\" (UniqueName: \"kubernetes.io/projected/5041edee-ad23-43bf-a84e-9bb6632c2c0b-kube-api-access-ftsb4\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-f2qb5\" (UID: \"5041edee-ad23-43bf-a84e-9bb6632c2c0b\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-f2qb5"
Oct 30 22:12:18.046076 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:18.046045    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hqm8\" (UniqueName: \"kubernetes.io/projected/06ae4cf6-b143-4fc7-91e7-69487df8809b-kube-api-access-8hqm8\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-m2jqj\" (UID: \"06ae4cf6-b143-4fc7-91e7-69487df8809b\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-m2jqj"
Oct 30 22:12:18.046242 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:18.046099    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06ae4cf6-b143-4fc7-91e7-69487df8809b-workload-certs\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-m2jqj\" (UID: \"06ae4cf6-b143-4fc7-91e7-69487df8809b\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-m2jqj"
Oct 30 22:12:18.046242 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:18.046197    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06ae4cf6-b143-4fc7-91e7-69487df8809b-istio-podinfo\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-m2jqj\" (UID: \"06ae4cf6-b143-4fc7-91e7-69487df8809b\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-m2jqj"
Oct 30 22:12:18.046242 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:18.046240    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06ae4cf6-b143-4fc7-91e7-69487df8809b-istio-data\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-m2jqj\" (UID: \"06ae4cf6-b143-4fc7-91e7-69487df8809b\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-m2jqj"
Oct 30 22:12:18.046477 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:18.046460    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06ae4cf6-b143-4fc7-91e7-69487df8809b-workload-socket\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-m2jqj\" (UID: \"06ae4cf6-b143-4fc7-91e7-69487df8809b\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-m2jqj"
Oct 30 22:12:18.046597 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:18.046583    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06ae4cf6-b143-4fc7-91e7-69487df8809b-istio-token\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-m2jqj\" (UID: \"06ae4cf6-b143-4fc7-91e7-69487df8809b\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-m2jqj"
Oct 30 22:12:18.046659 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:18.046630    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06ae4cf6-b143-4fc7-91e7-69487df8809b-credential-socket\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-m2jqj\" (UID: \"06ae4cf6-b143-4fc7-91e7-69487df8809b\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-m2jqj"
Oct 30 22:12:18.046707 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:18.046658    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06ae4cf6-b143-4fc7-91e7-69487df8809b-istio-envoy\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-m2jqj\" (UID: \"06ae4cf6-b143-4fc7-91e7-69487df8809b\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-m2jqj"
Oct 30 22:12:18.046707 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:18.046688    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06ae4cf6-b143-4fc7-91e7-69487df8809b-istiod-ca-cert\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-m2jqj\" (UID: \"06ae4cf6-b143-4fc7-91e7-69487df8809b\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-m2jqj"
Oct 30 22:12:18.116317 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:18.116294    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-f2qb5"
Oct 30 22:12:18.148322 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:18.148290    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06ae4cf6-b143-4fc7-91e7-69487df8809b-istio-token\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-m2jqj\" (UID: \"06ae4cf6-b143-4fc7-91e7-69487df8809b\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-m2jqj"
Oct 30 22:12:18.148472 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:18.148353    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06ae4cf6-b143-4fc7-91e7-69487df8809b-credential-socket\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-m2jqj\" (UID: \"06ae4cf6-b143-4fc7-91e7-69487df8809b\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-m2jqj"
Oct 30 22:12:18.148472 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:18.148384    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06ae4cf6-b143-4fc7-91e7-69487df8809b-istio-envoy\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-m2jqj\" (UID: \"06ae4cf6-b143-4fc7-91e7-69487df8809b\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-m2jqj"
Oct 30 22:12:18.148813 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:18.148782    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06ae4cf6-b143-4fc7-91e7-69487df8809b-credential-socket\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-m2jqj\" (UID: \"06ae4cf6-b143-4fc7-91e7-69487df8809b\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-m2jqj"
Oct 30 22:12:18.149142 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:18.149119    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06ae4cf6-b143-4fc7-91e7-69487df8809b-istiod-ca-cert\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-m2jqj\" (UID: \"06ae4cf6-b143-4fc7-91e7-69487df8809b\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-m2jqj"
Oct 30 22:12:18.149249 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:18.149233    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hqm8\" (UniqueName: \"kubernetes.io/projected/06ae4cf6-b143-4fc7-91e7-69487df8809b-kube-api-access-8hqm8\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-m2jqj\" (UID: \"06ae4cf6-b143-4fc7-91e7-69487df8809b\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-m2jqj"
Oct 30 22:12:18.149310 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:18.149277    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06ae4cf6-b143-4fc7-91e7-69487df8809b-workload-certs\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-m2jqj\" (UID: \"06ae4cf6-b143-4fc7-91e7-69487df8809b\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-m2jqj"
Oct 30 22:12:18.149363 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:18.149330    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06ae4cf6-b143-4fc7-91e7-69487df8809b-istio-podinfo\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-m2jqj\" (UID: \"06ae4cf6-b143-4fc7-91e7-69487df8809b\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-m2jqj"
Oct 30 22:12:18.149429 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:18.149378    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06ae4cf6-b143-4fc7-91e7-69487df8809b-istio-data\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-m2jqj\" (UID: \"06ae4cf6-b143-4fc7-91e7-69487df8809b\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-m2jqj"
Oct 30 22:12:18.149557 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:18.149536    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06ae4cf6-b143-4fc7-91e7-69487df8809b-workload-socket\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-m2jqj\" (UID: \"06ae4cf6-b143-4fc7-91e7-69487df8809b\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-m2jqj"
Oct 30 22:12:18.150025 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:18.150002    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06ae4cf6-b143-4fc7-91e7-69487df8809b-workload-socket\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-m2jqj\" (UID: \"06ae4cf6-b143-4fc7-91e7-69487df8809b\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-m2jqj"
Oct 30 22:12:18.150526 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:18.150503    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06ae4cf6-b143-4fc7-91e7-69487df8809b-workload-certs\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-m2jqj\" (UID: \"06ae4cf6-b143-4fc7-91e7-69487df8809b\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-m2jqj"
Oct 30 22:12:18.150718 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:18.150696    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06ae4cf6-b143-4fc7-91e7-69487df8809b-istio-data\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-m2jqj\" (UID: \"06ae4cf6-b143-4fc7-91e7-69487df8809b\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-m2jqj"
Oct 30 22:12:18.151038 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:18.151015    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06ae4cf6-b143-4fc7-91e7-69487df8809b-istiod-ca-cert\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-m2jqj\" (UID: \"06ae4cf6-b143-4fc7-91e7-69487df8809b\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-m2jqj"
Oct 30 22:12:18.152217 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:18.152192    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06ae4cf6-b143-4fc7-91e7-69487df8809b-istio-envoy\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-m2jqj\" (UID: \"06ae4cf6-b143-4fc7-91e7-69487df8809b\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-m2jqj"
Oct 30 22:12:18.152815 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:18.152794    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06ae4cf6-b143-4fc7-91e7-69487df8809b-istio-podinfo\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-m2jqj\" (UID: \"06ae4cf6-b143-4fc7-91e7-69487df8809b\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-m2jqj"
Oct 30 22:12:18.207776 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:18.207748    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06ae4cf6-b143-4fc7-91e7-69487df8809b-istio-token\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-m2jqj\" (UID: \"06ae4cf6-b143-4fc7-91e7-69487df8809b\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-m2jqj"
Oct 30 22:12:18.208807 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:18.208783    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hqm8\" (UniqueName: \"kubernetes.io/projected/06ae4cf6-b143-4fc7-91e7-69487df8809b-kube-api-access-8hqm8\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-m2jqj\" (UID: \"06ae4cf6-b143-4fc7-91e7-69487df8809b\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-m2jqj"
Oct 30 22:12:18.244785 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:18.244756    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-m2jqj"
Oct 30 22:12:18.560352 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:12:18.560326    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5041edee_ad23_43bf_a84e_9bb6632c2c0b.slice/crio-e4e58202088d7da9ecf9394102bf55a90e501e65e36836b567a63a9387b8ed51 WatchSource:0}: Error finding container e4e58202088d7da9ecf9394102bf55a90e501e65e36836b567a63a9387b8ed51: Status 404 returned error can't find the container with id e4e58202088d7da9ecf9394102bf55a90e501e65e36836b567a63a9387b8ed51
Oct 30 22:12:18.561320 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:18.561300    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-f2qb5"]
Oct 30 22:12:18.562970 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:18.562938    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:12:18.563076 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:18.563020    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:12:18.563076 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:18.563070    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:12:18.588535 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:18.588510    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-f2qb5" event={"ID":"5041edee-ad23-43bf-a84e-9bb6632c2c0b","Type":"ContainerStarted","Data":"e4e58202088d7da9ecf9394102bf55a90e501e65e36836b567a63a9387b8ed51"}
Oct 30 22:12:18.669890 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:12:18.669861    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06ae4cf6_b143_4fc7_91e7_69487df8809b.slice/crio-9316adb8049175d98105879f22d894a734d0880613c0bc76c986f8861d83c3d4 WatchSource:0}: Error finding container 9316adb8049175d98105879f22d894a734d0880613c0bc76c986f8861d83c3d4: Status 404 returned error can't find the container with id 9316adb8049175d98105879f22d894a734d0880613c0bc76c986f8861d83c3d4
Oct 30 22:12:18.670588 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:18.670565    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-m2jqj"]
Oct 30 22:12:18.672883 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:18.672851    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:12:18.672965 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:18.672919    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:12:18.672965 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:18.672958    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:12:19.598368 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:19.598330    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-f2qb5" event={"ID":"5041edee-ad23-43bf-a84e-9bb6632c2c0b","Type":"ContainerStarted","Data":"25ad9f97a8fe5ee3c92c23e1e4730013643b77854b77e606209453c96d9cd173"}
Oct 30 22:12:19.599935 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:19.599907    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-m2jqj" event={"ID":"06ae4cf6-b143-4fc7-91e7-69487df8809b","Type":"ContainerStarted","Data":"20d0b11ef9f5ffee8ad1c9094456d9e0654666826f5eb4d6f8d2591ba16e9241"}
Oct 30 22:12:19.600068 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:19.599951    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-m2jqj" event={"ID":"06ae4cf6-b143-4fc7-91e7-69487df8809b","Type":"ContainerStarted","Data":"9316adb8049175d98105879f22d894a734d0880613c0bc76c986f8861d83c3d4"}
Oct 30 22:12:19.777954 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:19.777890    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-f2qb5" podStartSLOduration=2.777870061 podStartE2EDuration="2.777870061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:19.773605321 +0000 UTC m=+1317.377912886" watchObservedRunningTime="2026-10-30 22:12:19.777870061 +0000 UTC m=+1317.382177615"
Oct 30 22:12:19.916105 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:19.916052    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-m2jqj" podStartSLOduration=2.916038136 podStartE2EDuration="2.916038136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:19.913042204 +0000 UTC m=+1317.517349755" watchObservedRunningTime="2026-10-30 22:12:19.916038136 +0000 UTC m=+1317.520345686"
Oct 30 22:12:19.976474 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:19.972535    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e62b0e13-td5c4"]
Oct 30 22:12:19.980532 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:19.979796    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e62b0e13-td5c4"
Oct 30 22:12:19.983154 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:19.983006    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69be4be8-fc7a-452d-859c-3740d226f614-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e62b0e13-td5c4\" (UID: \"69be4be8-fc7a-452d-859c-3740d226f614\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e62b0e13-td5c4"
Oct 30 22:12:19.983154 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:19.983058    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbxvz\" (UniqueName: \"kubernetes.io/projected/69be4be8-fc7a-452d-859c-3740d226f614-kube-api-access-bbxvz\") pod \"maas-api-revoke-sub-e62b0e13-td5c4\" (UID: \"69be4be8-fc7a-452d-859c-3740d226f614\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e62b0e13-td5c4"
Oct 30 22:12:19.983154 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:19.983135    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69be4be8-fc7a-452d-859c-3740d226f614-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e62b0e13-td5c4\" (UID: \"69be4be8-fc7a-452d-859c-3740d226f614\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e62b0e13-td5c4"
Oct 30 22:12:20.018849 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:20.018819    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e62b0e13-td5c4"]
Oct 30 22:12:20.043424 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:20.034714    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:12:20.043424 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:20.034883    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5rvcj\""
Oct 30 22:12:20.085427 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:20.085121    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69be4be8-fc7a-452d-859c-3740d226f614-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e62b0e13-td5c4\" (UID: \"69be4be8-fc7a-452d-859c-3740d226f614\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e62b0e13-td5c4"
Oct 30 22:12:20.085427 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:20.085202    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbxvz\" (UniqueName: \"kubernetes.io/projected/69be4be8-fc7a-452d-859c-3740d226f614-kube-api-access-bbxvz\") pod \"maas-api-revoke-sub-e62b0e13-td5c4\" (UID: \"69be4be8-fc7a-452d-859c-3740d226f614\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e62b0e13-td5c4"
Oct 30 22:12:20.085427 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:20.085252    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69be4be8-fc7a-452d-859c-3740d226f614-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e62b0e13-td5c4\" (UID: \"69be4be8-fc7a-452d-859c-3740d226f614\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e62b0e13-td5c4"
Oct 30 22:12:20.086507 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:20.086455    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69be4be8-fc7a-452d-859c-3740d226f614-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e62b0e13-td5c4\" (UID: \"69be4be8-fc7a-452d-859c-3740d226f614\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e62b0e13-td5c4"
Oct 30 22:12:20.088412 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:20.088376    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69be4be8-fc7a-452d-859c-3740d226f614-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e62b0e13-td5c4\" (UID: \"69be4be8-fc7a-452d-859c-3740d226f614\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e62b0e13-td5c4"
Oct 30 22:12:20.117431 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:20.117347    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-f2qb5"
Oct 30 22:12:20.118883 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:20.118654    5678 patch_prober.go:29] interesting pod/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-f2qb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Oct 30 22:12:20.118883 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:20.118700    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-f2qb5" podUID="5041edee-ad23-43bf-a84e-9bb6632c2c0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Oct 30 22:12:20.178331 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:20.178219    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbxvz\" (UniqueName: \"kubernetes.io/projected/69be4be8-fc7a-452d-859c-3740d226f614-kube-api-access-bbxvz\") pod \"maas-api-revoke-sub-e62b0e13-td5c4\" (UID: \"69be4be8-fc7a-452d-859c-3740d226f614\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e62b0e13-td5c4"
Oct 30 22:12:20.262425 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:20.254575    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-m2jqj"
Oct 30 22:12:20.262425 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:20.256749    5678 patch_prober.go:29] interesting pod/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-m2jqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Oct 30 22:12:20.262425 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:20.256812    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-m2jqj" podUID="06ae4cf6-b143-4fc7-91e7-69487df8809b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Oct 30 22:12:20.297457 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:20.297431    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e62b0e13-td5c4"
Oct 30 22:12:20.773948 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:12:20.773780    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69be4be8_fc7a_452d_859c_3740d226f614.slice/crio-0a33c1a7352b4d228243bd429cfe918384114a5dc34e00f1d3212f649fab0f2e WatchSource:0}: Error finding container 0a33c1a7352b4d228243bd429cfe918384114a5dc34e00f1d3212f649fab0f2e: Status 404 returned error can't find the container with id 0a33c1a7352b4d228243bd429cfe918384114a5dc34e00f1d3212f649fab0f2e
Oct 30 22:12:20.777685 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:20.777662    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e62b0e13-td5c4"]
Oct 30 22:12:21.117547 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:21.117522    5678 patch_prober.go:29] interesting pod/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-f2qb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Oct 30 22:12:21.117751 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:21.117731    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-f2qb5" podUID="5041edee-ad23-43bf-a84e-9bb6632c2c0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Oct 30 22:12:21.259436 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:21.257375    5678 patch_prober.go:29] interesting pod/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-m2jqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Oct 30 22:12:21.259436 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:21.257450    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-m2jqj" podUID="06ae4cf6-b143-4fc7-91e7-69487df8809b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Oct 30 22:12:21.617600 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:21.617553    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e62b0e13-td5c4" event={"ID":"69be4be8-fc7a-452d-859c-3740d226f614","Type":"ContainerStarted","Data":"3398adbaad00cbf191e24ca0549b8b2c6fc0a6524c1c6afb54b44f6b73851297"}
Oct 30 22:12:21.617767 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:21.617616    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e62b0e13-td5c4" event={"ID":"69be4be8-fc7a-452d-859c-3740d226f614","Type":"ContainerStarted","Data":"0a33c1a7352b4d228243bd429cfe918384114a5dc34e00f1d3212f649fab0f2e"}
Oct 30 22:12:22.116862 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:22.116829    5678 patch_prober.go:29] interesting pod/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-f2qb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Oct 30 22:12:22.117242 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:22.116873    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-f2qb5" podUID="5041edee-ad23-43bf-a84e-9bb6632c2c0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Oct 30 22:12:22.273864 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:22.273754    5678 patch_prober.go:29] interesting pod/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-m2jqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Oct 30 22:12:22.273864 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:22.273844    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-m2jqj" podUID="06ae4cf6-b143-4fc7-91e7-69487df8809b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Oct 30 22:12:22.653328 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:22.653291    5678 generic.go:356] "Generic (PLEG): container finished" podID="69be4be8-fc7a-452d-859c-3740d226f614" containerID="3398adbaad00cbf191e24ca0549b8b2c6fc0a6524c1c6afb54b44f6b73851297" exitCode=0
Oct 30 22:12:22.658078 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:22.657981    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e62b0e13-td5c4" event={"ID":"69be4be8-fc7a-452d-859c-3740d226f614","Type":"ContainerDied","Data":"3398adbaad00cbf191e24ca0549b8b2c6fc0a6524c1c6afb54b44f6b73851297"}
Oct 30 22:12:23.116675 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:23.116644    5678 patch_prober.go:29] interesting pod/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-f2qb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Oct 30 22:12:23.116825 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:23.116699    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-f2qb5" podUID="5041edee-ad23-43bf-a84e-9bb6632c2c0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Oct 30 22:12:23.257425 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:23.248854    5678 patch_prober.go:29] interesting pod/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-m2jqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Oct 30 22:12:23.257425 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:23.248926    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-m2jqj" podUID="06ae4cf6-b143-4fc7-91e7-69487df8809b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Oct 30 22:12:23.989265 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:23.987055    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e62b0e13-td5c4"
Oct 30 22:12:24.093624 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:24.093594    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69be4be8-fc7a-452d-859c-3740d226f614-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69be4be8-fc7a-452d-859c-3740d226f614-maas-api-openshift-ca\") pod \"69be4be8-fc7a-452d-859c-3740d226f614\" (UID: \"69be4be8-fc7a-452d-859c-3740d226f614\") "
Oct 30 22:12:24.093802 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:24.093730    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69be4be8-fc7a-452d-859c-3740d226f614-kube-api-access-bbxvz\" (UniqueName: \"kubernetes.io/projected/69be4be8-fc7a-452d-859c-3740d226f614-kube-api-access-bbxvz\") pod \"69be4be8-fc7a-452d-859c-3740d226f614\" (UID: \"69be4be8-fc7a-452d-859c-3740d226f614\") "
Oct 30 22:12:24.093802 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:24.093767    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/69be4be8-fc7a-452d-859c-3740d226f614-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69be4be8-fc7a-452d-859c-3740d226f614-maas-api-xks-ca\") pod \"69be4be8-fc7a-452d-859c-3740d226f614\" (UID: \"69be4be8-fc7a-452d-859c-3740d226f614\") "
Oct 30 22:12:24.093940 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:24.093916    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69be4be8-fc7a-452d-859c-3740d226f614-maas-api-openshift-ca" pod "69be4be8-fc7a-452d-859c-3740d226f614" (UID: "69be4be8-fc7a-452d-859c-3740d226f614"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:12:24.094395 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:24.094375    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69be4be8-fc7a-452d-859c-3740d226f614-maas-api-openshift-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:12:24.095934 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:24.095913    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69be4be8-fc7a-452d-859c-3740d226f614-maas-api-xks-ca" pod "69be4be8-fc7a-452d-859c-3740d226f614" (UID: "69be4be8-fc7a-452d-859c-3740d226f614"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:12:24.096007 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:24.095964    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69be4be8-fc7a-452d-859c-3740d226f614-kube-api-access-bbxvz" pod "69be4be8-fc7a-452d-859c-3740d226f614" (UID: "69be4be8-fc7a-452d-859c-3740d226f614"). InnerVolumeSpecName "kube-api-access-bbxvz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:12:24.116723 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:24.116699    5678 patch_prober.go:29] interesting pod/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-f2qb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Oct 30 22:12:24.116856 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:24.116748    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-f2qb5" podUID="5041edee-ad23-43bf-a84e-9bb6632c2c0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Oct 30 22:12:24.195802 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:24.195721    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbxvz\" (UniqueName: \"kubernetes.io/projected/69be4be8-fc7a-452d-859c-3740d226f614-kube-api-access-bbxvz\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:12:24.195802 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:24.195752    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69be4be8-fc7a-452d-859c-3740d226f614-maas-api-xks-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:12:24.247894 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:24.247859    5678 patch_prober.go:29] interesting pod/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-m2jqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Oct 30 22:12:24.248179 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:24.248130    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-m2jqj" podUID="06ae4cf6-b143-4fc7-91e7-69487df8809b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Oct 30 22:12:24.695314 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:24.695278    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e62b0e13-td5c4"
Oct 30 22:12:24.695754 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:24.695274    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e62b0e13-td5c4" event={"ID":"69be4be8-fc7a-452d-859c-3740d226f614","Type":"ContainerDied","Data":"0a33c1a7352b4d228243bd429cfe918384114a5dc34e00f1d3212f649fab0f2e"}
Oct 30 22:12:24.695754 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:24.695459    5678 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0a33c1a7352b4d228243bd429cfe918384114a5dc34e00f1d3212f649fab0f2e"
Oct 30 22:12:25.117295 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:25.117261    5678 patch_prober.go:29] interesting pod/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-f2qb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Oct 30 22:12:25.117486 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:25.117315    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-f2qb5" podUID="5041edee-ad23-43bf-a84e-9bb6632c2c0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Oct 30 22:12:25.246158 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:25.246124    5678 patch_prober.go:29] interesting pod/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-m2jqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Oct 30 22:12:25.246311 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:25.246179    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-m2jqj" podUID="06ae4cf6-b143-4fc7-91e7-69487df8809b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Oct 30 22:12:26.122511 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:26.122480    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-f2qb5"
Oct 30 22:12:26.123067 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:26.123045    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-f2qb5"
Oct 30 22:12:26.123992 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:26.123970    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-f2qb5"
Oct 30 22:12:26.249634 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:26.249605    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-m2jqj"
Oct 30 22:12:26.249896 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:26.249877    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-m2jqj"
Oct 30 22:12:26.250487 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:26.250464    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-m2jqj"
Oct 30 22:12:30.847552 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:30.847513    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b4f5ba7-4drxr"]
Oct 30 22:12:30.848490 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:30.848464    5678 cpu_manager.go:414] "containerMap: removing container" podUID="69be4be8-fc7a-452d-859c-3740d226f614" containerName="revoke-keys"
Oct 30 22:12:30.848490 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:30.848489    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69be4be8-fc7a-452d-859c-3740d226f614" containerName="revoke-keys"
Oct 30 22:12:30.848742 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:30.848725    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="69be4be8-fc7a-452d-859c-3740d226f614" containerName="revoke-keys"
Oct 30 22:12:30.853236 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:30.853211    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b4f5ba7-4drxr"
Oct 30 22:12:30.910655 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:30.910627    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b4f5ba7-4drxr"]
Oct 30 22:12:30.965554 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:30.965523    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5rvcj\""
Oct 30 22:12:30.965855 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:30.965837    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:12:30.990740 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:30.990372    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c68c37eb-ff02-4b46-ac99-6e8b081e091b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b4f5ba7-4drxr\" (UID: \"c68c37eb-ff02-4b46-ac99-6e8b081e091b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b4f5ba7-4drxr"
Oct 30 22:12:30.990740 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:30.990486    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c68c37eb-ff02-4b46-ac99-6e8b081e091b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b4f5ba7-4drxr\" (UID: \"c68c37eb-ff02-4b46-ac99-6e8b081e091b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b4f5ba7-4drxr"
Oct 30 22:12:30.990740 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:30.990620    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blc78\" (UniqueName: \"kubernetes.io/projected/c68c37eb-ff02-4b46-ac99-6e8b081e091b-kube-api-access-blc78\") pod \"maas-api-revoke-sub-1b4f5ba7-4drxr\" (UID: \"c68c37eb-ff02-4b46-ac99-6e8b081e091b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b4f5ba7-4drxr"
Oct 30 22:12:31.092184 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:31.092145    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blc78\" (UniqueName: \"kubernetes.io/projected/c68c37eb-ff02-4b46-ac99-6e8b081e091b-kube-api-access-blc78\") pod \"maas-api-revoke-sub-1b4f5ba7-4drxr\" (UID: \"c68c37eb-ff02-4b46-ac99-6e8b081e091b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b4f5ba7-4drxr"
Oct 30 22:12:31.092381 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:31.092278    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c68c37eb-ff02-4b46-ac99-6e8b081e091b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b4f5ba7-4drxr\" (UID: \"c68c37eb-ff02-4b46-ac99-6e8b081e091b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b4f5ba7-4drxr"
Oct 30 22:12:31.092637 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:31.092607    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c68c37eb-ff02-4b46-ac99-6e8b081e091b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b4f5ba7-4drxr\" (UID: \"c68c37eb-ff02-4b46-ac99-6e8b081e091b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b4f5ba7-4drxr"
Oct 30 22:12:31.093130 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:31.093066    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c68c37eb-ff02-4b46-ac99-6e8b081e091b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b4f5ba7-4drxr\" (UID: \"c68c37eb-ff02-4b46-ac99-6e8b081e091b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b4f5ba7-4drxr"
Oct 30 22:12:31.095228 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:31.095198    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c68c37eb-ff02-4b46-ac99-6e8b081e091b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b4f5ba7-4drxr\" (UID: \"c68c37eb-ff02-4b46-ac99-6e8b081e091b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b4f5ba7-4drxr"
Oct 30 22:12:31.174497 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:31.167841    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blc78\" (UniqueName: \"kubernetes.io/projected/c68c37eb-ff02-4b46-ac99-6e8b081e091b-kube-api-access-blc78\") pod \"maas-api-revoke-sub-1b4f5ba7-4drxr\" (UID: \"c68c37eb-ff02-4b46-ac99-6e8b081e091b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b4f5ba7-4drxr"
Oct 30 22:12:31.174497 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:31.173855    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b4f5ba7-4drxr"
Oct 30 22:12:31.609969 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:31.609917    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b4f5ba7-4drxr"]
Oct 30 22:12:31.610113 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:12:31.610070    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc68c37eb_ff02_4b46_ac99_6e8b081e091b.slice/crio-122f1396d8d562cb76288292d07ea582e863e3c6a5cb4958f61266f7d37f922c WatchSource:0}: Error finding container 122f1396d8d562cb76288292d07ea582e863e3c6a5cb4958f61266f7d37f922c: Status 404 returned error can't find the container with id 122f1396d8d562cb76288292d07ea582e863e3c6a5cb4958f61266f7d37f922c
Oct 30 22:12:31.752607 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:31.752576    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b4f5ba7-4drxr" event={"ID":"c68c37eb-ff02-4b46-ac99-6e8b081e091b","Type":"ContainerStarted","Data":"122f1396d8d562cb76288292d07ea582e863e3c6a5cb4958f61266f7d37f922c"}
Oct 30 22:12:32.761822 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:32.761738    5678 generic.go:356] "Generic (PLEG): container finished" podID="c68c37eb-ff02-4b46-ac99-6e8b081e091b" containerID="87a3cab586aba9c2299193f92f3212b6ef910b5c1314a583a113ec14b83aaf46" exitCode=0
Oct 30 22:12:32.762169 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:32.761818    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b4f5ba7-4drxr" event={"ID":"c68c37eb-ff02-4b46-ac99-6e8b081e091b","Type":"ContainerDied","Data":"87a3cab586aba9c2299193f92f3212b6ef910b5c1314a583a113ec14b83aaf46"}
Oct 30 22:12:33.957190 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:33.957163    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b4f5ba7-4drxr"
Oct 30 22:12:34.055843 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:34.044583    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c68c37eb-ff02-4b46-ac99-6e8b081e091b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c68c37eb-ff02-4b46-ac99-6e8b081e091b-maas-api-xks-ca\") pod \"c68c37eb-ff02-4b46-ac99-6e8b081e091b\" (UID: \"c68c37eb-ff02-4b46-ac99-6e8b081e091b\") "
Oct 30 22:12:34.055843 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:34.044662    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c68c37eb-ff02-4b46-ac99-6e8b081e091b-kube-api-access-blc78\" (UniqueName: \"kubernetes.io/projected/c68c37eb-ff02-4b46-ac99-6e8b081e091b-kube-api-access-blc78\") pod \"c68c37eb-ff02-4b46-ac99-6e8b081e091b\" (UID: \"c68c37eb-ff02-4b46-ac99-6e8b081e091b\") "
Oct 30 22:12:34.055843 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:34.044786    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c68c37eb-ff02-4b46-ac99-6e8b081e091b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c68c37eb-ff02-4b46-ac99-6e8b081e091b-maas-api-openshift-ca\") pod \"c68c37eb-ff02-4b46-ac99-6e8b081e091b\" (UID: \"c68c37eb-ff02-4b46-ac99-6e8b081e091b\") "
Oct 30 22:12:34.055843 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:34.051724    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c68c37eb-ff02-4b46-ac99-6e8b081e091b-maas-api-xks-ca" pod "c68c37eb-ff02-4b46-ac99-6e8b081e091b" (UID: "c68c37eb-ff02-4b46-ac99-6e8b081e091b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:12:34.055843 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:34.055539    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c68c37eb-ff02-4b46-ac99-6e8b081e091b-kube-api-access-blc78" pod "c68c37eb-ff02-4b46-ac99-6e8b081e091b" (UID: "c68c37eb-ff02-4b46-ac99-6e8b081e091b"). InnerVolumeSpecName "kube-api-access-blc78". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:12:34.055843 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:34.055724    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c68c37eb-ff02-4b46-ac99-6e8b081e091b-maas-api-openshift-ca" pod "c68c37eb-ff02-4b46-ac99-6e8b081e091b" (UID: "c68c37eb-ff02-4b46-ac99-6e8b081e091b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:12:34.147246 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:34.147196    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c68c37eb-ff02-4b46-ac99-6e8b081e091b-maas-api-openshift-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:12:34.147572 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:34.147531    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c68c37eb-ff02-4b46-ac99-6e8b081e091b-maas-api-xks-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:12:34.148135 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:34.148113    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blc78\" (UniqueName: \"kubernetes.io/projected/c68c37eb-ff02-4b46-ac99-6e8b081e091b-kube-api-access-blc78\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:12:34.777136 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:34.777110    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b4f5ba7-4drxr"
Oct 30 22:12:34.777298 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:34.777133    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b4f5ba7-4drxr" event={"ID":"c68c37eb-ff02-4b46-ac99-6e8b081e091b","Type":"ContainerDied","Data":"122f1396d8d562cb76288292d07ea582e863e3c6a5cb4958f61266f7d37f922c"}
Oct 30 22:12:34.777298 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:34.777183    5678 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="122f1396d8d562cb76288292d07ea582e863e3c6a5cb4958f61266f7d37f922c"
Oct 30 22:12:40.572557 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:12:40.572522    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 22:12:47.673305 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:47.673267    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-65e59f/distinct-2-w4-65e59f-kserve-689566757d-bf98q"]
Oct 30 22:12:47.674006 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:47.673989    5678 cpu_manager.go:414] "containerMap: removing container" podUID="c68c37eb-ff02-4b46-ac99-6e8b081e091b" containerName="revoke-keys"
Oct 30 22:12:47.674006 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:47.674007    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c68c37eb-ff02-4b46-ac99-6e8b081e091b" containerName="revoke-keys"
Oct 30 22:12:47.674137 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:47.674128    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="c68c37eb-ff02-4b46-ac99-6e8b081e091b" containerName="revoke-keys"
Oct 30 22:12:47.677742 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:47.677724    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-65e59f/distinct-2-w4-65e59f-kserve-689566757d-bf98q"
Oct 30 22:12:47.733782 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:47.733744    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64e66fdb-056e-4086-a85a-8c02613584a5-home\") pod \"distinct-2-w4-65e59f-kserve-689566757d-bf98q\" (UID: \"64e66fdb-056e-4086-a85a-8c02613584a5\") " pod="e2e-models-e2e-worker-w4-65e59f/distinct-2-w4-65e59f-kserve-689566757d-bf98q"
Oct 30 22:12:47.733996 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:47.733819    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64e66fdb-056e-4086-a85a-8c02613584a5-tmp-dir\") pod \"distinct-2-w4-65e59f-kserve-689566757d-bf98q\" (UID: \"64e66fdb-056e-4086-a85a-8c02613584a5\") " pod="e2e-models-e2e-worker-w4-65e59f/distinct-2-w4-65e59f-kserve-689566757d-bf98q"
Oct 30 22:12:47.733996 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:47.733912    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64e66fdb-056e-4086-a85a-8c02613584a5-tls-certs\") pod \"distinct-2-w4-65e59f-kserve-689566757d-bf98q\" (UID: \"64e66fdb-056e-4086-a85a-8c02613584a5\") " pod="e2e-models-e2e-worker-w4-65e59f/distinct-2-w4-65e59f-kserve-689566757d-bf98q"
Oct 30 22:12:47.734119 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:47.734051    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-np7vq\" (UniqueName: \"kubernetes.io/projected/64e66fdb-056e-4086-a85a-8c02613584a5-kube-api-access-np7vq\") pod \"distinct-2-w4-65e59f-kserve-689566757d-bf98q\" (UID: \"64e66fdb-056e-4086-a85a-8c02613584a5\") " pod="e2e-models-e2e-worker-w4-65e59f/distinct-2-w4-65e59f-kserve-689566757d-bf98q"
Oct 30 22:12:47.734190 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:47.734114    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64e66fdb-056e-4086-a85a-8c02613584a5-dshm\") pod \"distinct-2-w4-65e59f-kserve-689566757d-bf98q\" (UID: \"64e66fdb-056e-4086-a85a-8c02613584a5\") " pod="e2e-models-e2e-worker-w4-65e59f/distinct-2-w4-65e59f-kserve-689566757d-bf98q"
Oct 30 22:12:47.734315 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:47.734299    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64e66fdb-056e-4086-a85a-8c02613584a5-model-cache\") pod \"distinct-2-w4-65e59f-kserve-689566757d-bf98q\" (UID: \"64e66fdb-056e-4086-a85a-8c02613584a5\") " pod="e2e-models-e2e-worker-w4-65e59f/distinct-2-w4-65e59f-kserve-689566757d-bf98q"
Oct 30 22:12:47.775959 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:47.775933    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-65e59f\"/\"distinct-2-w4-65e59f-kserve-self-signed-certs\""
Oct 30 22:12:47.780609 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:47.780587    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-65e59f/distinct-2-w4-65e59f-kserve-689566757d-bf98q"]
Oct 30 22:12:47.835237 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:47.835208    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-np7vq\" (UniqueName: \"kubernetes.io/projected/64e66fdb-056e-4086-a85a-8c02613584a5-kube-api-access-np7vq\") pod \"distinct-2-w4-65e59f-kserve-689566757d-bf98q\" (UID: \"64e66fdb-056e-4086-a85a-8c02613584a5\") " pod="e2e-models-e2e-worker-w4-65e59f/distinct-2-w4-65e59f-kserve-689566757d-bf98q"
Oct 30 22:12:47.835436 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:47.835259    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64e66fdb-056e-4086-a85a-8c02613584a5-dshm\") pod \"distinct-2-w4-65e59f-kserve-689566757d-bf98q\" (UID: \"64e66fdb-056e-4086-a85a-8c02613584a5\") " pod="e2e-models-e2e-worker-w4-65e59f/distinct-2-w4-65e59f-kserve-689566757d-bf98q"
Oct 30 22:12:47.835436 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:47.835290    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64e66fdb-056e-4086-a85a-8c02613584a5-model-cache\") pod \"distinct-2-w4-65e59f-kserve-689566757d-bf98q\" (UID: \"64e66fdb-056e-4086-a85a-8c02613584a5\") " pod="e2e-models-e2e-worker-w4-65e59f/distinct-2-w4-65e59f-kserve-689566757d-bf98q"
Oct 30 22:12:47.835436 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:47.835350    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64e66fdb-056e-4086-a85a-8c02613584a5-home\") pod \"distinct-2-w4-65e59f-kserve-689566757d-bf98q\" (UID: \"64e66fdb-056e-4086-a85a-8c02613584a5\") " pod="e2e-models-e2e-worker-w4-65e59f/distinct-2-w4-65e59f-kserve-689566757d-bf98q"
Oct 30 22:12:47.835436 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:47.835391    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64e66fdb-056e-4086-a85a-8c02613584a5-tmp-dir\") pod \"distinct-2-w4-65e59f-kserve-689566757d-bf98q\" (UID: \"64e66fdb-056e-4086-a85a-8c02613584a5\") " pod="e2e-models-e2e-worker-w4-65e59f/distinct-2-w4-65e59f-kserve-689566757d-bf98q"
Oct 30 22:12:47.835679 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:47.835503    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64e66fdb-056e-4086-a85a-8c02613584a5-tls-certs\") pod \"distinct-2-w4-65e59f-kserve-689566757d-bf98q\" (UID: \"64e66fdb-056e-4086-a85a-8c02613584a5\") " pod="e2e-models-e2e-worker-w4-65e59f/distinct-2-w4-65e59f-kserve-689566757d-bf98q"
Oct 30 22:12:47.835810 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:47.835788    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64e66fdb-056e-4086-a85a-8c02613584a5-model-cache\") pod \"distinct-2-w4-65e59f-kserve-689566757d-bf98q\" (UID: \"64e66fdb-056e-4086-a85a-8c02613584a5\") " pod="e2e-models-e2e-worker-w4-65e59f/distinct-2-w4-65e59f-kserve-689566757d-bf98q"
Oct 30 22:12:47.835866 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:47.835849    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64e66fdb-056e-4086-a85a-8c02613584a5-home\") pod \"distinct-2-w4-65e59f-kserve-689566757d-bf98q\" (UID: \"64e66fdb-056e-4086-a85a-8c02613584a5\") " pod="e2e-models-e2e-worker-w4-65e59f/distinct-2-w4-65e59f-kserve-689566757d-bf98q"
Oct 30 22:12:47.835906 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:47.835859    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64e66fdb-056e-4086-a85a-8c02613584a5-tmp-dir\") pod \"distinct-2-w4-65e59f-kserve-689566757d-bf98q\" (UID: \"64e66fdb-056e-4086-a85a-8c02613584a5\") " pod="e2e-models-e2e-worker-w4-65e59f/distinct-2-w4-65e59f-kserve-689566757d-bf98q"
Oct 30 22:12:47.837675 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:47.837653    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64e66fdb-056e-4086-a85a-8c02613584a5-dshm\") pod \"distinct-2-w4-65e59f-kserve-689566757d-bf98q\" (UID: \"64e66fdb-056e-4086-a85a-8c02613584a5\") " pod="e2e-models-e2e-worker-w4-65e59f/distinct-2-w4-65e59f-kserve-689566757d-bf98q"
Oct 30 22:12:47.837927 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:47.837911    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64e66fdb-056e-4086-a85a-8c02613584a5-tls-certs\") pod \"distinct-2-w4-65e59f-kserve-689566757d-bf98q\" (UID: \"64e66fdb-056e-4086-a85a-8c02613584a5\") " pod="e2e-models-e2e-worker-w4-65e59f/distinct-2-w4-65e59f-kserve-689566757d-bf98q"
Oct 30 22:12:47.894591 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:47.894557    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-np7vq\" (UniqueName: \"kubernetes.io/projected/64e66fdb-056e-4086-a85a-8c02613584a5-kube-api-access-np7vq\") pod \"distinct-2-w4-65e59f-kserve-689566757d-bf98q\" (UID: \"64e66fdb-056e-4086-a85a-8c02613584a5\") " pod="e2e-models-e2e-worker-w4-65e59f/distinct-2-w4-65e59f-kserve-689566757d-bf98q"
Oct 30 22:12:47.996197 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:47.996113    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-65e59f/distinct-2-w4-65e59f-kserve-689566757d-bf98q"
Oct 30 22:12:48.414708 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:12:48.414678    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod64e66fdb_056e_4086_a85a_8c02613584a5.slice/crio-da2545beef0e2cb33fe413da44cc8ad4fd599e875f3e77d35fa2cd47b0cb9cb4 WatchSource:0}: Error finding container da2545beef0e2cb33fe413da44cc8ad4fd599e875f3e77d35fa2cd47b0cb9cb4: Status 404 returned error can't find the container with id da2545beef0e2cb33fe413da44cc8ad4fd599e875f3e77d35fa2cd47b0cb9cb4
Oct 30 22:12:48.414873 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:48.414853    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-65e59f/distinct-2-w4-65e59f-kserve-689566757d-bf98q"]
Oct 30 22:12:48.899498 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:48.899458    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-65e59f/distinct-2-w4-65e59f-kserve-689566757d-bf98q" event={"ID":"64e66fdb-056e-4086-a85a-8c02613584a5","Type":"ContainerStarted","Data":"f3733e4c04c8dc60bd942cdf795c540ecb4b8b621d3f6debc73088508c8880b9"}
Oct 30 22:12:48.899902 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:48.899513    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-65e59f/distinct-2-w4-65e59f-kserve-689566757d-bf98q" event={"ID":"64e66fdb-056e-4086-a85a-8c02613584a5","Type":"ContainerStarted","Data":"da2545beef0e2cb33fe413da44cc8ad4fd599e875f3e77d35fa2cd47b0cb9cb4"}
Oct 30 22:12:48.996563 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:48.996534    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-65e59f/distinct-2-w4-65e59f-kserve-689566757d-bf98q"
Oct 30 22:12:49.013634 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:49.013583    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-65e59f/distinct-2-w4-65e59f-kserve-689566757d-bf98q" podStartSLOduration=2.013570904 podStartE2EDuration="2.013570904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:49.010664955 +0000 UTC m=+1346.614972508" watchObservedRunningTime="2026-10-30 22:12:49.013570904 +0000 UTC m=+1346.617878457"
Oct 30 22:12:49.187015 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:49.186928    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34a1fc25-hmlzl"]
Oct 30 22:12:49.192130 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:49.192108    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34a1fc25-hmlzl"
Oct 30 22:12:49.241981 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:49.241936    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:12:49.242151 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:49.242069    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5rvcj\""
Oct 30 22:12:49.245135 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:49.245112    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34a1fc25-hmlzl"]
Oct 30 22:12:49.252923 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:49.252897    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvnwc\" (UniqueName: \"kubernetes.io/projected/66b4ab22-b3fa-4ed8-9ade-370752cf2041-kube-api-access-jvnwc\") pod \"maas-api-revoke-sub-34a1fc25-hmlzl\" (UID: \"66b4ab22-b3fa-4ed8-9ade-370752cf2041\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34a1fc25-hmlzl"
Oct 30 22:12:49.253048 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:49.252940    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66b4ab22-b3fa-4ed8-9ade-370752cf2041-maas-api-xks-ca\") pod \"maas-api-revoke-sub-34a1fc25-hmlzl\" (UID: \"66b4ab22-b3fa-4ed8-9ade-370752cf2041\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34a1fc25-hmlzl"
Oct 30 22:12:49.253208 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:49.253183    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66b4ab22-b3fa-4ed8-9ade-370752cf2041-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-34a1fc25-hmlzl\" (UID: \"66b4ab22-b3fa-4ed8-9ade-370752cf2041\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34a1fc25-hmlzl"
Oct 30 22:12:49.354754 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:49.354718    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66b4ab22-b3fa-4ed8-9ade-370752cf2041-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-34a1fc25-hmlzl\" (UID: \"66b4ab22-b3fa-4ed8-9ade-370752cf2041\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34a1fc25-hmlzl"
Oct 30 22:12:49.354946 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:49.354928    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvnwc\" (UniqueName: \"kubernetes.io/projected/66b4ab22-b3fa-4ed8-9ade-370752cf2041-kube-api-access-jvnwc\") pod \"maas-api-revoke-sub-34a1fc25-hmlzl\" (UID: \"66b4ab22-b3fa-4ed8-9ade-370752cf2041\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34a1fc25-hmlzl"
Oct 30 22:12:49.355018 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:49.354962    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66b4ab22-b3fa-4ed8-9ade-370752cf2041-maas-api-xks-ca\") pod \"maas-api-revoke-sub-34a1fc25-hmlzl\" (UID: \"66b4ab22-b3fa-4ed8-9ade-370752cf2041\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34a1fc25-hmlzl"
Oct 30 22:12:49.355491 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:49.355465    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66b4ab22-b3fa-4ed8-9ade-370752cf2041-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-34a1fc25-hmlzl\" (UID: \"66b4ab22-b3fa-4ed8-9ade-370752cf2041\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34a1fc25-hmlzl"
Oct 30 22:12:49.357194 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:49.357173    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66b4ab22-b3fa-4ed8-9ade-370752cf2041-maas-api-xks-ca\") pod \"maas-api-revoke-sub-34a1fc25-hmlzl\" (UID: \"66b4ab22-b3fa-4ed8-9ade-370752cf2041\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34a1fc25-hmlzl"
Oct 30 22:12:49.414108 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:49.414079    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvnwc\" (UniqueName: \"kubernetes.io/projected/66b4ab22-b3fa-4ed8-9ade-370752cf2041-kube-api-access-jvnwc\") pod \"maas-api-revoke-sub-34a1fc25-hmlzl\" (UID: \"66b4ab22-b3fa-4ed8-9ade-370752cf2041\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34a1fc25-hmlzl"
Oct 30 22:12:49.506226 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:49.506151    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34a1fc25-hmlzl"
Oct 30 22:12:49.900303 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:49.900250    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34a1fc25-hmlzl"]
Oct 30 22:12:49.901139 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:12:49.901114    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66b4ab22_b3fa_4ed8_9ade_370752cf2041.slice/crio-7417c96c61a3742ce7a2b3c768c7d375477c9bde71d5fd24a62e1b53a774ab1e WatchSource:0}: Error finding container 7417c96c61a3742ce7a2b3c768c7d375477c9bde71d5fd24a62e1b53a774ab1e: Status 404 returned error can't find the container with id 7417c96c61a3742ce7a2b3c768c7d375477c9bde71d5fd24a62e1b53a774ab1e
Oct 30 22:12:50.917048 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:50.917010    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34a1fc25-hmlzl" event={"ID":"66b4ab22-b3fa-4ed8-9ade-370752cf2041","Type":"ContainerStarted","Data":"af848e644c9f307a49e16c1978ef8f5c3b042a5bd829c9195b1c5dbb18ad49fa"}
Oct 30 22:12:50.917459 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:50.917066    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34a1fc25-hmlzl" event={"ID":"66b4ab22-b3fa-4ed8-9ade-370752cf2041","Type":"ContainerStarted","Data":"7417c96c61a3742ce7a2b3c768c7d375477c9bde71d5fd24a62e1b53a774ab1e"}
Oct 30 22:12:51.034636 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:51.034474    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34a1fc25-hmlzl" podStartSLOduration=2.034454535 podStartE2EDuration="2.034454535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:51.029134278 +0000 UTC m=+1348.633441834" watchObservedRunningTime="2026-10-30 22:12:51.034454535 +0000 UTC m=+1348.638762090"
Oct 30 22:12:51.934837 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:51.934800    5678 generic.go:356] "Generic (PLEG): container finished" podID="66b4ab22-b3fa-4ed8-9ade-370752cf2041" containerID="af848e644c9f307a49e16c1978ef8f5c3b042a5bd829c9195b1c5dbb18ad49fa" exitCode=0
Oct 30 22:12:51.935231 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:51.934885    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34a1fc25-hmlzl" event={"ID":"66b4ab22-b3fa-4ed8-9ade-370752cf2041","Type":"ContainerDied","Data":"af848e644c9f307a49e16c1978ef8f5c3b042a5bd829c9195b1c5dbb18ad49fa"}
Oct 30 22:12:53.096308 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:53.096283    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34a1fc25-hmlzl"
Oct 30 22:12:53.209460 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:53.209365    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66b4ab22-b3fa-4ed8-9ade-370752cf2041-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66b4ab22-b3fa-4ed8-9ade-370752cf2041-maas-api-xks-ca\") pod \"66b4ab22-b3fa-4ed8-9ade-370752cf2041\" (UID: \"66b4ab22-b3fa-4ed8-9ade-370752cf2041\") "
Oct 30 22:12:53.209460 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:53.209433    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66b4ab22-b3fa-4ed8-9ade-370752cf2041-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66b4ab22-b3fa-4ed8-9ade-370752cf2041-maas-api-openshift-ca\") pod \"66b4ab22-b3fa-4ed8-9ade-370752cf2041\" (UID: \"66b4ab22-b3fa-4ed8-9ade-370752cf2041\") "
Oct 30 22:12:53.209745 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:53.209570    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66b4ab22-b3fa-4ed8-9ade-370752cf2041-kube-api-access-jvnwc\" (UniqueName: \"kubernetes.io/projected/66b4ab22-b3fa-4ed8-9ade-370752cf2041-kube-api-access-jvnwc\") pod \"66b4ab22-b3fa-4ed8-9ade-370752cf2041\" (UID: \"66b4ab22-b3fa-4ed8-9ade-370752cf2041\") "
Oct 30 22:12:53.209864 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:53.209840    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66b4ab22-b3fa-4ed8-9ade-370752cf2041-maas-api-openshift-ca" pod "66b4ab22-b3fa-4ed8-9ade-370752cf2041" (UID: "66b4ab22-b3fa-4ed8-9ade-370752cf2041"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:12:53.210411 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:53.210364    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66b4ab22-b3fa-4ed8-9ade-370752cf2041-maas-api-openshift-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:12:53.211928 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:53.211904    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66b4ab22-b3fa-4ed8-9ade-370752cf2041-maas-api-xks-ca" pod "66b4ab22-b3fa-4ed8-9ade-370752cf2041" (UID: "66b4ab22-b3fa-4ed8-9ade-370752cf2041"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:12:53.212815 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:53.212790    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66b4ab22-b3fa-4ed8-9ade-370752cf2041-kube-api-access-jvnwc" pod "66b4ab22-b3fa-4ed8-9ade-370752cf2041" (UID: "66b4ab22-b3fa-4ed8-9ade-370752cf2041"). InnerVolumeSpecName "kube-api-access-jvnwc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:12:53.311776 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:53.311739    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66b4ab22-b3fa-4ed8-9ade-370752cf2041-maas-api-xks-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:12:53.311776 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:53.311771    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvnwc\" (UniqueName: \"kubernetes.io/projected/66b4ab22-b3fa-4ed8-9ade-370752cf2041-kube-api-access-jvnwc\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:12:53.952583 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:53.952558    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34a1fc25-hmlzl"
Oct 30 22:12:53.952583 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:53.952565    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34a1fc25-hmlzl" event={"ID":"66b4ab22-b3fa-4ed8-9ade-370752cf2041","Type":"ContainerDied","Data":"7417c96c61a3742ce7a2b3c768c7d375477c9bde71d5fd24a62e1b53a774ab1e"}
Oct 30 22:12:53.952822 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:53.952624    5678 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7417c96c61a3742ce7a2b3c768c7d375477c9bde71d5fd24a62e1b53a774ab1e"
Oct 30 22:12:57.996845 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:57.996812    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-65e59f/distinct-2-w4-65e59f-kserve-689566757d-bf98q"
Oct 30 22:12:58.009535 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:58.009512    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-65e59f/distinct-2-w4-65e59f-kserve-689566757d-bf98q"
Oct 30 22:12:59.007811 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:12:59.007783    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-65e59f/distinct-2-w4-65e59f-kserve-689566757d-bf98q"
Oct 30 22:13:01.033564 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:01.033507    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-65e59f/simulator-w4-65e59f-kserve-85698bcd47-przch" podUID="ee5e843b-17be-4623-8c95-dc07c8b7f506" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:13:01.094979 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:01.094935    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-826abc20-4zllk"]
Oct 30 22:13:01.105752 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:01.105643    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb0b93/distinct-2-w6-fb0b93-kserve-7d65bb4698-m6tkh" podUID="6fd276ef-522b-4bb4-b101-0a9c3fe9ecd3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:13:01.142594 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:01.142348    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-826abc20-4zllk"]
Oct 30 22:13:01.190054 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:01.190009    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779f32/premium-w0-779f32-kserve-67978797c8-jpkvl" podUID="40138fb9-9a75-4458-a351-11eb61c4a1e3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Oct 30 22:13:01.237564 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:01.237513    5678 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-65e59f/premium-w4-65e59f-kserve-6dc49468f8-rlzhx" podUID="8590a783-eb24-4759-ad74-1369f36aa9d2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:13:01.338626 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:01.338520    5678 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-fb0b93/simulator-w6-fb0b93-kserve-55b784b479-588bn" podUID="951a6f7e-a22e-40f2-ace2-1ce5d3bc9b49" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:13:01.494560 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:01.494492    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a00d9b/distinct-2-w1-a00d9b-kserve-68fd95957c-dmgnb" podUID="23a81fda-f7e3-4f91-8d96-9f19c7de6e09" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:13:01.502516 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:01.502472    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9ndd9" podUID="81bd6415-f42f-446b-a64e-956f794b3956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:13:01.503029 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:01.503002    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7bb1c2/premium-w3-7bb1c2-kserve-7dbf875d88-dqxn5" podUID="a58c911f-e2eb-4820-a6b6-24afe78b5fd4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:13:01.541534 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:01.541485    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-65e59f/premium-w4-65e59f-kserve-6dc49468f8-rlzhx" podUID="8590a783-eb24-4759-ad74-1369f36aa9d2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded"
Oct 30 22:13:01.706509 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:01.706466    5678 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-fb0b93/distinct-2-w6-fb0b93-kserve-7d65bb4698-m6tkh" podUID="6fd276ef-522b-4bb4-b101-0a9c3fe9ecd3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:13:01.825660 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:01.825599    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb0b93/simulator-w6-fb0b93-kserve-55b784b479-588bn" podUID="951a6f7e-a22e-40f2-ace2-1ce5d3bc9b49" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:13:01.950598 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:01.950543    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7bb1c2/simulator-w3-7bb1c2-kserve-675645869c-vcwqv" podUID="1fce9daf-68c9-41d0-825b-2012bb93c8ba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:13:02.014606 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:02.013507    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-65e59f/distinct-2-w4-65e59f-kserve-689566757d-bf98q" podUID="64e66fdb-056e-4086-a85a-8c02613584a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:13:02.040598 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:02.038517    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-65e59f/simulator-w4-65e59f-kserve-85698bcd47-przch" podUID="ee5e843b-17be-4623-8c95-dc07c8b7f506" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:13:02.040598 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:02.038948    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-65e59f/simulator-w4-65e59f-kserve-85698bcd47-przch"
Oct 30 22:13:02.117545 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:02.117501    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb0b93/distinct-2-w6-fb0b93-kserve-7d65bb4698-m6tkh" podUID="6fd276ef-522b-4bb4-b101-0a9c3fe9ecd3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:13:02.118196 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:02.118172    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-fb0b93/distinct-2-w6-fb0b93-kserve-7d65bb4698-m6tkh"
Oct 30 22:13:02.191422 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:02.190477    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779f32/premium-w0-779f32-kserve-67978797c8-jpkvl" podUID="40138fb9-9a75-4458-a351-11eb61c4a1e3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Oct 30 22:13:02.191422 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:02.190877    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-779f32/premium-w0-779f32-kserve-67978797c8-jpkvl"
Oct 30 22:13:02.311635 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:02.311539    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-jjlvw" podUID="67f802f4-805b-4503-88f7-31a0da8e6172" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.134.0.28:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:13:02.334430 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:02.319482    5678 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a00d9b/distinct-2-w1-a00d9b-kserve-68fd95957c-dmgnb" podUID="23a81fda-f7e3-4f91-8d96-9f19c7de6e09" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:13:02.542491 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:02.531270    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9ndd9" podUID="81bd6415-f42f-446b-a64e-956f794b3956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": context deadline exceeded"
Oct 30 22:13:02.542491 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:02.539491    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7bb1c2/premium-w3-7bb1c2-kserve-7dbf875d88-dqxn5" podUID="a58c911f-e2eb-4820-a6b6-24afe78b5fd4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Oct 30 22:13:02.567609 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:02.555965    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-65e59f/premium-w4-65e59f-kserve-6dc49468f8-rlzhx" podUID="8590a783-eb24-4759-ad74-1369f36aa9d2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:13:02.567609 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:02.556324    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9ndd9"
Oct 30 22:13:02.567609 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:02.556390    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-7bb1c2/premium-w3-7bb1c2-kserve-7dbf875d88-dqxn5"
Oct 30 22:13:02.567609 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:02.556446    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-65e59f/premium-w4-65e59f-kserve-6dc49468f8-rlzhx"
Oct 30 22:13:02.570426 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:02.568181    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a00d9b/distinct-2-w1-a00d9b-kserve-68fd95957c-dmgnb" podUID="23a81fda-f7e3-4f91-8d96-9f19c7de6e09" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded"
Oct 30 22:13:02.570426 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:02.568420    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a00d9b/distinct-2-w1-a00d9b-kserve-68fd95957c-dmgnb"
Oct 30 22:13:02.665442 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:02.661502    5678 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-7bb1c2/distinct-w3-7bb1c2-kserve-8d695596c-8zgtw" podUID="36f0a6a2-54be-49bf-9202-502960cbb134" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": context deadline exceeded"
Oct 30 22:13:02.688458 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:02.685435    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7bb1c2/distinct-w3-7bb1c2-kserve-8d695596c-8zgtw" podUID="36f0a6a2-54be-49bf-9202-502960cbb134" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:13:02.802449 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:02.793500    5678 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-7bb1c2/simulator-w3-7bb1c2-kserve-675645869c-vcwqv" podUID="1fce9daf-68c9-41d0-825b-2012bb93c8ba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:13:02.847483 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:02.830495    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb0b93/simulator-w6-fb0b93-kserve-55b784b479-588bn" podUID="951a6f7e-a22e-40f2-ace2-1ce5d3bc9b49" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 30 22:13:02.916425 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:02.915536    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="938761fa-a709-4b8b-ba91-5caf62caeee2" path="/var/lib/kubelet/pods/938761fa-a709-4b8b-ba91-5caf62caeee2/volumes"
Oct 30 22:13:02.924434 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:02.922735    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-fb0b93/simulator-w6-fb0b93-kserve-55b784b479-588bn"
Oct 30 22:13:02.953557 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:02.953506    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7bb1c2/simulator-w3-7bb1c2-kserve-675645869c-vcwqv" podUID="1fce9daf-68c9-41d0-825b-2012bb93c8ba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Oct 30 22:13:02.978474 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:02.978444    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-7bb1c2/simulator-w3-7bb1c2-kserve-675645869c-vcwqv"
Oct 30 22:13:03.014543 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:03.014494    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-65e59f/distinct-2-w4-65e59f-kserve-689566757d-bf98q" podUID="64e66fdb-056e-4086-a85a-8c02613584a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:13:03.015031 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:03.015009    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-65e59f/distinct-2-w4-65e59f-kserve-689566757d-bf98q"
Oct 30 22:13:03.040538 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:03.040500    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-65e59f/simulator-w4-65e59f-kserve-85698bcd47-przch" podUID="ee5e843b-17be-4623-8c95-dc07c8b7f506" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:13:03.366622 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:03.240870    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779f32/premium-w0-779f32-kserve-67978797c8-jpkvl" podUID="40138fb9-9a75-4458-a351-11eb61c4a1e3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:13:03.381568 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:03.306955    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb0b93/distinct-2-w6-fb0b93-kserve-7d65bb4698-m6tkh" podUID="6fd276ef-522b-4bb4-b101-0a9c3fe9ecd3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": context deadline exceeded"
Oct 30 22:13:03.541621 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:03.541581    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7bb1c2/premium-w3-7bb1c2-kserve-7dbf875d88-dqxn5" podUID="a58c911f-e2eb-4820-a6b6-24afe78b5fd4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:13:03.542363 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:03.542317    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9ndd9" podUID="81bd6415-f42f-446b-a64e-956f794b3956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": context deadline exceeded"
Oct 30 22:13:03.582864 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:03.582810    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a00d9b/distinct-2-w1-a00d9b-kserve-68fd95957c-dmgnb" podUID="23a81fda-f7e3-4f91-8d96-9f19c7de6e09" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded"
Oct 30 22:13:03.602445 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:03.600488    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-65e59f/premium-w4-65e59f-kserve-6dc49468f8-rlzhx" podUID="8590a783-eb24-4759-ad74-1369f36aa9d2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded"
Oct 30 22:13:03.678471 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:03.673337    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-65e59f/simulator-w4-65e59f-kserve-85698bcd47-przch"
Oct 30 22:13:03.697283 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:03.692371    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7bb1c2/distinct-w3-7bb1c2-kserve-8d695596c-8zgtw" podUID="36f0a6a2-54be-49bf-9202-502960cbb134" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:13:03.697283 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:03.692749    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-7bb1c2/distinct-w3-7bb1c2-kserve-8d695596c-8zgtw"
Oct 30 22:13:03.929498 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:03.919488    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb0b93/simulator-w6-fb0b93-kserve-55b784b479-588bn" podUID="951a6f7e-a22e-40f2-ace2-1ce5d3bc9b49" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:13:03.936341 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:03.934892    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9ndd9"
Oct 30 22:13:03.936341 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:03.935099    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-7bb1c2/premium-w3-7bb1c2-kserve-7dbf875d88-dqxn5"
Oct 30 22:13:03.937710 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:03.937350    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-7bb1c2/distinct-w3-7bb1c2-kserve-8d695596c-8zgtw"
Oct 30 22:13:03.946687 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:03.945831    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a00d9b/distinct-2-w1-a00d9b-kserve-68fd95957c-dmgnb"
Oct 30 22:13:03.962067 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:03.962043    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-fb0b93/distinct-2-w6-fb0b93-kserve-7d65bb4698-m6tkh"
Oct 30 22:13:03.981502 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:03.981472    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7bb1c2/simulator-w3-7bb1c2-kserve-675645869c-vcwqv" podUID="1fce9daf-68c9-41d0-825b-2012bb93c8ba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Oct 30 22:13:03.998880 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:03.993344    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-65e59f/distinct-2-w4-65e59f-kserve-689566757d-bf98q"
Oct 30 22:13:03.999533 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:03.999515    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-779f32/premium-w0-779f32-kserve-67978797c8-jpkvl"
Oct 30 22:13:04.007629 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:04.007609    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-65e59f/premium-w4-65e59f-kserve-6dc49468f8-rlzhx"
Oct 30 22:13:04.048263 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:04.048207    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-7bb1c2/simulator-w3-7bb1c2-kserve-675645869c-vcwqv"
Oct 30 22:13:04.057895 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:04.057869    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-fb0b93/simulator-w6-fb0b93-kserve-55b784b479-588bn"
Oct 30 22:13:05.795953 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:05.795912    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1515f79a-wn8hw"]
Oct 30 22:13:05.796786 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:05.796763    5678 cpu_manager.go:414] "containerMap: removing container" podUID="66b4ab22-b3fa-4ed8-9ade-370752cf2041" containerName="revoke-keys"
Oct 30 22:13:05.796900 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:05.796787    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66b4ab22-b3fa-4ed8-9ade-370752cf2041" containerName="revoke-keys"
Oct 30 22:13:05.796978 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:05.796965    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="66b4ab22-b3fa-4ed8-9ade-370752cf2041" containerName="revoke-keys"
Oct 30 22:13:05.801746 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:05.801723    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1515f79a-wn8hw"
Oct 30 22:13:05.854135 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:05.854111    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:13:05.854273 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:05.854162    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5rvcj\""
Oct 30 22:13:05.858109 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:05.858082    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1515f79a-wn8hw"]
Oct 30 22:13:05.889181 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:05.889144    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24484902-a1b7-488b-bfcc-666e79cc88ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1515f79a-wn8hw\" (UID: \"24484902-a1b7-488b-bfcc-666e79cc88ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1515f79a-wn8hw"
Oct 30 22:13:05.889181 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:05.889176    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24484902-a1b7-488b-bfcc-666e79cc88ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1515f79a-wn8hw\" (UID: \"24484902-a1b7-488b-bfcc-666e79cc88ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1515f79a-wn8hw"
Oct 30 22:13:05.889383 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:05.889263    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d24l5\" (UniqueName: \"kubernetes.io/projected/24484902-a1b7-488b-bfcc-666e79cc88ee-kube-api-access-d24l5\") pod \"maas-api-revoke-sub-1515f79a-wn8hw\" (UID: \"24484902-a1b7-488b-bfcc-666e79cc88ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1515f79a-wn8hw"
Oct 30 22:13:05.990187 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:05.990153    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24484902-a1b7-488b-bfcc-666e79cc88ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1515f79a-wn8hw\" (UID: \"24484902-a1b7-488b-bfcc-666e79cc88ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1515f79a-wn8hw"
Oct 30 22:13:05.990187 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:05.990187    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24484902-a1b7-488b-bfcc-666e79cc88ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1515f79a-wn8hw\" (UID: \"24484902-a1b7-488b-bfcc-666e79cc88ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1515f79a-wn8hw"
Oct 30 22:13:05.990490 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:05.990226    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d24l5\" (UniqueName: \"kubernetes.io/projected/24484902-a1b7-488b-bfcc-666e79cc88ee-kube-api-access-d24l5\") pod \"maas-api-revoke-sub-1515f79a-wn8hw\" (UID: \"24484902-a1b7-488b-bfcc-666e79cc88ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1515f79a-wn8hw"
Oct 30 22:13:05.990908 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:05.990882    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24484902-a1b7-488b-bfcc-666e79cc88ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1515f79a-wn8hw\" (UID: \"24484902-a1b7-488b-bfcc-666e79cc88ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1515f79a-wn8hw"
Oct 30 22:13:05.992601 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:05.992582    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24484902-a1b7-488b-bfcc-666e79cc88ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1515f79a-wn8hw\" (UID: \"24484902-a1b7-488b-bfcc-666e79cc88ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1515f79a-wn8hw"
Oct 30 22:13:06.049907 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:06.049838    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d24l5\" (UniqueName: \"kubernetes.io/projected/24484902-a1b7-488b-bfcc-666e79cc88ee-kube-api-access-d24l5\") pod \"maas-api-revoke-sub-1515f79a-wn8hw\" (UID: \"24484902-a1b7-488b-bfcc-666e79cc88ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1515f79a-wn8hw"
Oct 30 22:13:06.115294 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:06.115267    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1515f79a-wn8hw"
Oct 30 22:13:06.508362 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:13:06.508331    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24484902_a1b7_488b_bfcc_666e79cc88ee.slice/crio-d98f87332a8513d58c727fae6da55836c265772c55b704c1d876f5c05daf91ac WatchSource:0}: Error finding container d98f87332a8513d58c727fae6da55836c265772c55b704c1d876f5c05daf91ac: Status 404 returned error can't find the container with id d98f87332a8513d58c727fae6da55836c265772c55b704c1d876f5c05daf91ac
Oct 30 22:13:06.508990 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:06.508959    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1515f79a-wn8hw"]
Oct 30 22:13:07.445264 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:07.445226    5678 generic.go:356] "Generic (PLEG): container finished" podID="24484902-a1b7-488b-bfcc-666e79cc88ee" containerID="0d50ec7ba68f313a6174199ad97d7a5732393f66af75db9ac9d5e5b029d9cf75" exitCode=0
Oct 30 22:13:07.445622 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:07.445284    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1515f79a-wn8hw" event={"ID":"24484902-a1b7-488b-bfcc-666e79cc88ee","Type":"ContainerDied","Data":"0d50ec7ba68f313a6174199ad97d7a5732393f66af75db9ac9d5e5b029d9cf75"}
Oct 30 22:13:07.445622 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:07.445352    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1515f79a-wn8hw" event={"ID":"24484902-a1b7-488b-bfcc-666e79cc88ee","Type":"ContainerStarted","Data":"d98f87332a8513d58c727fae6da55836c265772c55b704c1d876f5c05daf91ac"}
Oct 30 22:13:08.606422 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:08.606377    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1515f79a-wn8hw"
Oct 30 22:13:08.728303 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:08.728221    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24484902-a1b7-488b-bfcc-666e79cc88ee-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24484902-a1b7-488b-bfcc-666e79cc88ee-maas-api-openshift-ca\") pod \"24484902-a1b7-488b-bfcc-666e79cc88ee\" (UID: \"24484902-a1b7-488b-bfcc-666e79cc88ee\") "
Oct 30 22:13:08.728513 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:08.728329    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/24484902-a1b7-488b-bfcc-666e79cc88ee-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24484902-a1b7-488b-bfcc-666e79cc88ee-maas-api-xks-ca\") pod \"24484902-a1b7-488b-bfcc-666e79cc88ee\" (UID: \"24484902-a1b7-488b-bfcc-666e79cc88ee\") "
Oct 30 22:13:08.728513 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:08.728484    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24484902-a1b7-488b-bfcc-666e79cc88ee-kube-api-access-d24l5\" (UniqueName: \"kubernetes.io/projected/24484902-a1b7-488b-bfcc-666e79cc88ee-kube-api-access-d24l5\") pod \"24484902-a1b7-488b-bfcc-666e79cc88ee\" (UID: \"24484902-a1b7-488b-bfcc-666e79cc88ee\") "
Oct 30 22:13:08.728710 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:08.728683    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24484902-a1b7-488b-bfcc-666e79cc88ee-maas-api-openshift-ca" pod "24484902-a1b7-488b-bfcc-666e79cc88ee" (UID: "24484902-a1b7-488b-bfcc-666e79cc88ee"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:13:08.729159 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:08.729133    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24484902-a1b7-488b-bfcc-666e79cc88ee-maas-api-openshift-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:13:08.730563 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:08.730530    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24484902-a1b7-488b-bfcc-666e79cc88ee-kube-api-access-d24l5" pod "24484902-a1b7-488b-bfcc-666e79cc88ee" (UID: "24484902-a1b7-488b-bfcc-666e79cc88ee"). InnerVolumeSpecName "kube-api-access-d24l5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:13:08.730856 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:08.730834    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24484902-a1b7-488b-bfcc-666e79cc88ee-maas-api-xks-ca" pod "24484902-a1b7-488b-bfcc-666e79cc88ee" (UID: "24484902-a1b7-488b-bfcc-666e79cc88ee"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:13:08.830903 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:08.830873    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24484902-a1b7-488b-bfcc-666e79cc88ee-maas-api-xks-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:13:08.830903 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:08.830900    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d24l5\" (UniqueName: \"kubernetes.io/projected/24484902-a1b7-488b-bfcc-666e79cc88ee-kube-api-access-d24l5\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:13:09.461364 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:09.461313    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1515f79a-wn8hw" event={"ID":"24484902-a1b7-488b-bfcc-666e79cc88ee","Type":"ContainerDied","Data":"d98f87332a8513d58c727fae6da55836c265772c55b704c1d876f5c05daf91ac"}
Oct 30 22:13:09.461542 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:09.461387    5678 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d98f87332a8513d58c727fae6da55836c265772c55b704c1d876f5c05daf91ac"
Oct 30 22:13:09.461542 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:09.461328    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1515f79a-wn8hw"
Oct 30 22:13:10.572554 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:13:10.572518    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 22:13:17.819285 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:17.819252    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9bfdx"]
Oct 30 22:13:17.820064 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:17.820045    5678 cpu_manager.go:414] "containerMap: removing container" podUID="24484902-a1b7-488b-bfcc-666e79cc88ee" containerName="revoke-keys"
Oct 30 22:13:17.820115 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:17.820065    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24484902-a1b7-488b-bfcc-666e79cc88ee" containerName="revoke-keys"
Oct 30 22:13:17.820206 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:17.820197    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="24484902-a1b7-488b-bfcc-666e79cc88ee" containerName="revoke-keys"
Oct 30 22:13:17.823792 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:17.823773    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9bfdx"
Oct 30 22:13:17.956788 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:17.956754    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1b614bbd-09b5-4b79-a9a5-1c57ae795852-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9bfdx\" (UID: \"1b614bbd-09b5-4b79-a9a5-1c57ae795852\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9bfdx"
Oct 30 22:13:17.957003 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:17.956831    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1b614bbd-09b5-4b79-a9a5-1c57ae795852-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9bfdx\" (UID: \"1b614bbd-09b5-4b79-a9a5-1c57ae795852\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9bfdx"
Oct 30 22:13:17.957003 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:17.956890    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b614bbd-09b5-4b79-a9a5-1c57ae795852-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9bfdx\" (UID: \"1b614bbd-09b5-4b79-a9a5-1c57ae795852\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9bfdx"
Oct 30 22:13:17.957003 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:17.956960    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zn9pc\" (UniqueName: \"kubernetes.io/projected/1b614bbd-09b5-4b79-a9a5-1c57ae795852-kube-api-access-zn9pc\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9bfdx\" (UID: \"1b614bbd-09b5-4b79-a9a5-1c57ae795852\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9bfdx"
Oct 30 22:13:17.957003 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:17.956998    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b614bbd-09b5-4b79-a9a5-1c57ae795852-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9bfdx\" (UID: \"1b614bbd-09b5-4b79-a9a5-1c57ae795852\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9bfdx"
Oct 30 22:13:17.957202 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:17.957045    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1b614bbd-09b5-4b79-a9a5-1c57ae795852-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9bfdx\" (UID: \"1b614bbd-09b5-4b79-a9a5-1c57ae795852\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9bfdx"
Oct 30 22:13:17.957297 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:17.957276    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1b614bbd-09b5-4b79-a9a5-1c57ae795852-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9bfdx\" (UID: \"1b614bbd-09b5-4b79-a9a5-1c57ae795852\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9bfdx"
Oct 30 22:13:18.058175 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:18.058134    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1b614bbd-09b5-4b79-a9a5-1c57ae795852-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9bfdx\" (UID: \"1b614bbd-09b5-4b79-a9a5-1c57ae795852\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9bfdx"
Oct 30 22:13:18.058370 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:18.058204    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1b614bbd-09b5-4b79-a9a5-1c57ae795852-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9bfdx\" (UID: \"1b614bbd-09b5-4b79-a9a5-1c57ae795852\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9bfdx"
Oct 30 22:13:18.058370 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:18.058261    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b614bbd-09b5-4b79-a9a5-1c57ae795852-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9bfdx\" (UID: \"1b614bbd-09b5-4b79-a9a5-1c57ae795852\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9bfdx"
Oct 30 22:13:18.058370 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:18.058286    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zn9pc\" (UniqueName: \"kubernetes.io/projected/1b614bbd-09b5-4b79-a9a5-1c57ae795852-kube-api-access-zn9pc\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9bfdx\" (UID: \"1b614bbd-09b5-4b79-a9a5-1c57ae795852\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9bfdx"
Oct 30 22:13:18.058370 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:18.058327    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b614bbd-09b5-4b79-a9a5-1c57ae795852-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9bfdx\" (UID: \"1b614bbd-09b5-4b79-a9a5-1c57ae795852\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9bfdx"
Oct 30 22:13:18.058594 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:18.058389    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1b614bbd-09b5-4b79-a9a5-1c57ae795852-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9bfdx\" (UID: \"1b614bbd-09b5-4b79-a9a5-1c57ae795852\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9bfdx"
Oct 30 22:13:18.058594 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:18.058584    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1b614bbd-09b5-4b79-a9a5-1c57ae795852-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9bfdx\" (UID: \"1b614bbd-09b5-4b79-a9a5-1c57ae795852\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9bfdx"
Oct 30 22:13:18.059321 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:18.059290    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1b614bbd-09b5-4b79-a9a5-1c57ae795852-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9bfdx\" (UID: \"1b614bbd-09b5-4b79-a9a5-1c57ae795852\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9bfdx"
Oct 30 22:13:18.060684 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:18.060661    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b614bbd-09b5-4b79-a9a5-1c57ae795852-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9bfdx\" (UID: \"1b614bbd-09b5-4b79-a9a5-1c57ae795852\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9bfdx"
Oct 30 22:13:18.394244 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:18.394214    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 30 22:13:18.394244 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:18.394215    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 30 22:13:18.394244 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:18.394230    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 30 22:13:18.396111 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:18.396094    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-zl688"]
Oct 30 22:13:18.400495 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:18.400469    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-q828p"]
Oct 30 22:13:18.400632 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:18.400575    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-zl688"
Oct 30 22:13:18.401564 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:18.401546    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1b614bbd-09b5-4b79-a9a5-1c57ae795852-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9bfdx\" (UID: \"1b614bbd-09b5-4b79-a9a5-1c57ae795852\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9bfdx"
Oct 30 22:13:18.401707 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:18.401685    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1b614bbd-09b5-4b79-a9a5-1c57ae795852-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9bfdx\" (UID: \"1b614bbd-09b5-4b79-a9a5-1c57ae795852\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9bfdx"
Oct 30 22:13:18.401753 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:18.401685    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1b614bbd-09b5-4b79-a9a5-1c57ae795852-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9bfdx\" (UID: \"1b614bbd-09b5-4b79-a9a5-1c57ae795852\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9bfdx"
Oct 30 22:13:18.405754 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:18.405735    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-q828p"
Oct 30 22:13:18.445564 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:18.445540    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-wzxhd\""
Oct 30 22:13:18.456841 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:18.456813    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b614bbd-09b5-4b79-a9a5-1c57ae795852-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9bfdx\" (UID: \"1b614bbd-09b5-4b79-a9a5-1c57ae795852\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9bfdx"
Oct 30 22:13:18.456954 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:18.456932    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zn9pc\" (UniqueName: \"kubernetes.io/projected/1b614bbd-09b5-4b79-a9a5-1c57ae795852-kube-api-access-zn9pc\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9bfdx\" (UID: \"1b614bbd-09b5-4b79-a9a5-1c57ae795852\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9bfdx"
Oct 30 22:13:18.566561 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:18.566530    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6373802-bc1e-4606-b946-4ae9753c560a-istio-podinfo\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-zl688\" (UID: \"c6373802-bc1e-4606-b946-4ae9753c560a\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-zl688"
Oct 30 22:13:18.566719 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:18.566598    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e8c9108-98ce-4a09-a8a9-f13def9228df-istio-envoy\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-q828p\" (UID: \"4e8c9108-98ce-4a09-a8a9-f13def9228df\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-q828p"
Oct 30 22:13:18.566719 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:18.566709    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e8c9108-98ce-4a09-a8a9-f13def9228df-istio-podinfo\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-q828p\" (UID: \"4e8c9108-98ce-4a09-a8a9-f13def9228df\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-q828p"
Oct 30 22:13:18.566792 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:18.566766    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6373802-bc1e-4606-b946-4ae9753c560a-istio-envoy\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-zl688\" (UID: \"c6373802-bc1e-4606-b946-4ae9753c560a\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-zl688"
Oct 30 22:13:18.566845 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:18.566829    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6373802-bc1e-4606-b946-4ae9753c560a-workload-socket\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-zl688\" (UID: \"c6373802-bc1e-4606-b946-4ae9753c560a\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-zl688"
Oct 30 22:13:18.566882 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:18.566853    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6373802-bc1e-4606-b946-4ae9753c560a-istio-token\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-zl688\" (UID: \"c6373802-bc1e-4606-b946-4ae9753c560a\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-zl688"
Oct 30 22:13:18.566939 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:18.566926    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6373802-bc1e-4606-b946-4ae9753c560a-workload-certs\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-zl688\" (UID: \"c6373802-bc1e-4606-b946-4ae9753c560a\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-zl688"
Oct 30 22:13:18.566972 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:18.566948    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5z87k\" (UniqueName: \"kubernetes.io/projected/c6373802-bc1e-4606-b946-4ae9753c560a-kube-api-access-5z87k\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-zl688\" (UID: \"c6373802-bc1e-4606-b946-4ae9753c560a\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-zl688"
Oct 30 22:13:18.567020 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:18.567005    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e8c9108-98ce-4a09-a8a9-f13def9228df-istio-token\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-q828p\" (UID: \"4e8c9108-98ce-4a09-a8a9-f13def9228df\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-q828p"
Oct 30 22:13:18.567055 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:18.567036    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e8c9108-98ce-4a09-a8a9-f13def9228df-workload-socket\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-q828p\" (UID: \"4e8c9108-98ce-4a09-a8a9-f13def9228df\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-q828p"
Oct 30 22:13:18.567090 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:18.567076    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e8c9108-98ce-4a09-a8a9-f13def9228df-istio-data\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-q828p\" (UID: \"4e8c9108-98ce-4a09-a8a9-f13def9228df\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-q828p"
Oct 30 22:13:18.567124 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:18.567095    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6373802-bc1e-4606-b946-4ae9753c560a-istio-data\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-zl688\" (UID: \"c6373802-bc1e-4606-b946-4ae9753c560a\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-zl688"
Oct 30 22:13:18.567184 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:18.567171    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6373802-bc1e-4606-b946-4ae9753c560a-istiod-ca-cert\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-zl688\" (UID: \"c6373802-bc1e-4606-b946-4ae9753c560a\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-zl688"
Oct 30 22:13:18.567236 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:18.567202    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e8c9108-98ce-4a09-a8a9-f13def9228df-workload-certs\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-q828p\" (UID: \"4e8c9108-98ce-4a09-a8a9-f13def9228df\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-q828p"
Oct 30 22:13:18.567290 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:18.567251    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6373802-bc1e-4606-b946-4ae9753c560a-credential-socket\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-zl688\" (UID: \"c6373802-bc1e-4606-b946-4ae9753c560a\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-zl688"
Oct 30 22:13:18.567290 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:18.567286    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e8c9108-98ce-4a09-a8a9-f13def9228df-credential-socket\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-q828p\" (UID: \"4e8c9108-98ce-4a09-a8a9-f13def9228df\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-q828p"
Oct 30 22:13:18.567434 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:18.567314    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndlj2\" (UniqueName: \"kubernetes.io/projected/4e8c9108-98ce-4a09-a8a9-f13def9228df-kube-api-access-ndlj2\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-q828p\" (UID: \"4e8c9108-98ce-4a09-a8a9-f13def9228df\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-q828p"
Oct 30 22:13:18.567434 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:18.567349    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e8c9108-98ce-4a09-a8a9-f13def9228df-istiod-ca-cert\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-q828p\" (UID: \"4e8c9108-98ce-4a09-a8a9-f13def9228df\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-q828p"
Oct 30 22:13:18.669220 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:18.669141    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e8c9108-98ce-4a09-a8a9-f13def9228df-istio-podinfo\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-q828p\" (UID: \"4e8c9108-98ce-4a09-a8a9-f13def9228df\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-q828p"
Oct 30 22:13:18.669220 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:18.669203    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6373802-bc1e-4606-b946-4ae9753c560a-istio-envoy\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-zl688\" (UID: \"c6373802-bc1e-4606-b946-4ae9753c560a\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-zl688"
Oct 30 22:13:18.669549 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:18.669232    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6373802-bc1e-4606-b946-4ae9753c560a-workload-socket\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-zl688\" (UID: \"c6373802-bc1e-4606-b946-4ae9753c560a\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-zl688"
Oct 30 22:13:18.669549 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:18.669250    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6373802-bc1e-4606-b946-4ae9753c560a-istio-token\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-zl688\" (UID: \"c6373802-bc1e-4606-b946-4ae9753c560a\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-zl688"
Oct 30 22:13:18.669549 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:18.669272    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6373802-bc1e-4606-b946-4ae9753c560a-workload-certs\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-zl688\" (UID: \"c6373802-bc1e-4606-b946-4ae9753c560a\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-zl688"
Oct 30 22:13:18.669549 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:18.669288    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5z87k\" (UniqueName: \"kubernetes.io/projected/c6373802-bc1e-4606-b946-4ae9753c560a-kube-api-access-5z87k\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-zl688\" (UID: \"c6373802-bc1e-4606-b946-4ae9753c560a\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-zl688"
Oct 30 22:13:18.669549 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:18.669307    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e8c9108-98ce-4a09-a8a9-f13def9228df-istio-token\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-q828p\" (UID: \"4e8c9108-98ce-4a09-a8a9-f13def9228df\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-q828p"
Oct 30 22:13:18.669549 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:18.669435    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e8c9108-98ce-4a09-a8a9-f13def9228df-workload-socket\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-q828p\" (UID: \"4e8c9108-98ce-4a09-a8a9-f13def9228df\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-q828p"
Oct 30 22:13:18.669549 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:18.669493    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e8c9108-98ce-4a09-a8a9-f13def9228df-istio-data\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-q828p\" (UID: \"4e8c9108-98ce-4a09-a8a9-f13def9228df\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-q828p"
Oct 30 22:13:18.669549 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:18.669520    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6373802-bc1e-4606-b946-4ae9753c560a-istio-data\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-zl688\" (UID: \"c6373802-bc1e-4606-b946-4ae9753c560a\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-zl688"
Oct 30 22:13:18.669959 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:18.669639    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6373802-bc1e-4606-b946-4ae9753c560a-istiod-ca-cert\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-zl688\" (UID: \"c6373802-bc1e-4606-b946-4ae9753c560a\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-zl688"
Oct 30 22:13:18.669959 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:18.669677    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e8c9108-98ce-4a09-a8a9-f13def9228df-workload-certs\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-q828p\" (UID: \"4e8c9108-98ce-4a09-a8a9-f13def9228df\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-q828p"
Oct 30 22:13:18.669959 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:18.669711    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6373802-bc1e-4606-b946-4ae9753c560a-workload-socket\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-zl688\" (UID: \"c6373802-bc1e-4606-b946-4ae9753c560a\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-zl688"
Oct 30 22:13:18.669959 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:18.669717    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6373802-bc1e-4606-b946-4ae9753c560a-credential-socket\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-zl688\" (UID: \"c6373802-bc1e-4606-b946-4ae9753c560a\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-zl688"
Oct 30 22:13:18.669959 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:18.669727    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6373802-bc1e-4606-b946-4ae9753c560a-workload-certs\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-zl688\" (UID: \"c6373802-bc1e-4606-b946-4ae9753c560a\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-zl688"
Oct 30 22:13:18.669959 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:18.669796    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e8c9108-98ce-4a09-a8a9-f13def9228df-credential-socket\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-q828p\" (UID: \"4e8c9108-98ce-4a09-a8a9-f13def9228df\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-q828p"
Oct 30 22:13:18.669959 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:18.669862    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndlj2\" (UniqueName: \"kubernetes.io/projected/4e8c9108-98ce-4a09-a8a9-f13def9228df-kube-api-access-ndlj2\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-q828p\" (UID: \"4e8c9108-98ce-4a09-a8a9-f13def9228df\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-q828p"
Oct 30 22:13:18.669959 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:18.669906    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e8c9108-98ce-4a09-a8a9-f13def9228df-workload-socket\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-q828p\" (UID: \"4e8c9108-98ce-4a09-a8a9-f13def9228df\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-q828p"
Oct 30 22:13:18.669959 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:18.669911    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e8c9108-98ce-4a09-a8a9-f13def9228df-istiod-ca-cert\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-q828p\" (UID: \"4e8c9108-98ce-4a09-a8a9-f13def9228df\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-q828p"
Oct 30 22:13:18.669959 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:18.669931    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6373802-bc1e-4606-b946-4ae9753c560a-istio-data\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-zl688\" (UID: \"c6373802-bc1e-4606-b946-4ae9753c560a\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-zl688"
Oct 30 22:13:18.669959 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:18.669956    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e8c9108-98ce-4a09-a8a9-f13def9228df-istio-data\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-q828p\" (UID: \"4e8c9108-98ce-4a09-a8a9-f13def9228df\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-q828p"
Oct 30 22:13:18.670562 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:18.669975    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6373802-bc1e-4606-b946-4ae9753c560a-credential-socket\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-zl688\" (UID: \"c6373802-bc1e-4606-b946-4ae9753c560a\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-zl688"
Oct 30 22:13:18.670562 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:18.669984    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6373802-bc1e-4606-b946-4ae9753c560a-istio-podinfo\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-zl688\" (UID: \"c6373802-bc1e-4606-b946-4ae9753c560a\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-zl688"
Oct 30 22:13:18.670562 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:18.670115    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e8c9108-98ce-4a09-a8a9-f13def9228df-istio-envoy\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-q828p\" (UID: \"4e8c9108-98ce-4a09-a8a9-f13def9228df\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-q828p"
Oct 30 22:13:18.670562 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:18.670161    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e8c9108-98ce-4a09-a8a9-f13def9228df-workload-certs\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-q828p\" (UID: \"4e8c9108-98ce-4a09-a8a9-f13def9228df\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-q828p"
Oct 30 22:13:18.670562 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:18.670317    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e8c9108-98ce-4a09-a8a9-f13def9228df-credential-socket\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-q828p\" (UID: \"4e8c9108-98ce-4a09-a8a9-f13def9228df\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-q828p"
Oct 30 22:13:18.670767 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:18.670607    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6373802-bc1e-4606-b946-4ae9753c560a-istiod-ca-cert\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-zl688\" (UID: \"c6373802-bc1e-4606-b946-4ae9753c560a\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-zl688"
Oct 30 22:13:18.670767 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:18.670614    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e8c9108-98ce-4a09-a8a9-f13def9228df-istiod-ca-cert\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-q828p\" (UID: \"4e8c9108-98ce-4a09-a8a9-f13def9228df\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-q828p"
Oct 30 22:13:18.672229 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:18.672207    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e8c9108-98ce-4a09-a8a9-f13def9228df-istio-podinfo\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-q828p\" (UID: \"4e8c9108-98ce-4a09-a8a9-f13def9228df\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-q828p"
Oct 30 22:13:18.672324 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:18.672249    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e8c9108-98ce-4a09-a8a9-f13def9228df-istio-envoy\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-q828p\" (UID: \"4e8c9108-98ce-4a09-a8a9-f13def9228df\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-q828p"
Oct 30 22:13:18.672603 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:18.672586    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6373802-bc1e-4606-b946-4ae9753c560a-istio-envoy\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-zl688\" (UID: \"c6373802-bc1e-4606-b946-4ae9753c560a\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-zl688"
Oct 30 22:13:18.672805 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:18.672790    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6373802-bc1e-4606-b946-4ae9753c560a-istio-podinfo\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-zl688\" (UID: \"c6373802-bc1e-4606-b946-4ae9753c560a\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-zl688"
Oct 30 22:13:18.734841 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:18.734809    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e8c9108-98ce-4a09-a8a9-f13def9228df-istio-token\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-q828p\" (UID: \"4e8c9108-98ce-4a09-a8a9-f13def9228df\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-q828p"
Oct 30 22:13:18.736838 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:18.736817    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9bfdx"
Oct 30 22:13:18.742797 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:18.742765    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6373802-bc1e-4606-b946-4ae9753c560a-istio-token\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-zl688\" (UID: \"c6373802-bc1e-4606-b946-4ae9753c560a\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-zl688"
Oct 30 22:13:18.742973 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:18.742952    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndlj2\" (UniqueName: \"kubernetes.io/projected/4e8c9108-98ce-4a09-a8a9-f13def9228df-kube-api-access-ndlj2\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-q828p\" (UID: \"4e8c9108-98ce-4a09-a8a9-f13def9228df\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-q828p"
Oct 30 22:13:18.743113 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:18.743093    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5z87k\" (UniqueName: \"kubernetes.io/projected/c6373802-bc1e-4606-b946-4ae9753c560a-kube-api-access-5z87k\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-zl688\" (UID: \"c6373802-bc1e-4606-b946-4ae9753c560a\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-zl688"
Oct 30 22:13:18.744824 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:18.744804    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-q828p"
Oct 30 22:13:18.949392 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:18.949322    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9bfdx"]
Oct 30 22:13:18.972473 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:18.972438    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-q828p"]
Oct 30 22:13:18.978756 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:18.978728    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-zl688"]
Oct 30 22:13:19.038956 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:19.038788    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-zl688"
Oct 30 22:13:19.390393 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:19.390071    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9bfdx"]
Oct 30 22:13:19.390393 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:19.390306    5678 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 22:13:19.391926 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:19.391900    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:13:19.391996 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:19.391969    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:13:19.392052 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:19.391997    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:13:19.416191 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:19.415092    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-q828p"]
Oct 30 22:13:19.464911 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:19.460833    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-zl688"]
Oct 30 22:13:19.464911 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:13:19.461098    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6373802_bc1e_4606_b946_4ae9753c560a.slice/crio-340c4e245eab6f343fe1c3f3833d2bf2f410acf584bfc87e526e634863ccc337 WatchSource:0}: Error finding container 340c4e245eab6f343fe1c3f3833d2bf2f410acf584bfc87e526e634863ccc337: Status 404 returned error can't find the container with id 340c4e245eab6f343fe1c3f3833d2bf2f410acf584bfc87e526e634863ccc337
Oct 30 22:13:19.465033 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:19.464960    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:13:19.465086 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:19.465044    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:13:19.465131 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:19.465086    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:13:19.544008 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:19.543812    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-q828p" event={"ID":"4e8c9108-98ce-4a09-a8a9-f13def9228df","Type":"ContainerStarted","Data":"6850f4ac14eb5dfdc02469456f3c2aaae042e5fe7ab85bbc8fa45a39e4bd3435"}
Oct 30 22:13:19.544008 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:19.543879    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-q828p" event={"ID":"4e8c9108-98ce-4a09-a8a9-f13def9228df","Type":"ContainerStarted","Data":"3ae2b279a90bcbbe7afe2914dcdd6456942e608b059cedea1280d60fb7a7a285"}
Oct 30 22:13:19.546968 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:19.546862    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9bfdx" event={"ID":"1b614bbd-09b5-4b79-a9a5-1c57ae795852","Type":"ContainerStarted","Data":"33234d2431251f16d37364e70b09ff26814413778ecd249b29aebb8bcfd89afa"}
Oct 30 22:13:19.551686 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:19.551626    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-zl688" event={"ID":"c6373802-bc1e-4606-b946-4ae9753c560a","Type":"ContainerStarted","Data":"340c4e245eab6f343fe1c3f3833d2bf2f410acf584bfc87e526e634863ccc337"}
Oct 30 22:13:19.679270 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:19.679148    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-q828p" podStartSLOduration=2.679128384 podStartE2EDuration="2.679128384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:13:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:13:19.676958798 +0000 UTC m=+1377.281266354" watchObservedRunningTime="2026-10-30 22:13:19.679128384 +0000 UTC m=+1377.283435938"
Oct 30 22:13:19.745700 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:19.745659    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-q828p"
Oct 30 22:13:19.811043 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:19.810984    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-zl688" podStartSLOduration=2.810970153 podStartE2EDuration="2.810970153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:13:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:13:19.808666487 +0000 UTC m=+1377.412974041" watchObservedRunningTime="2026-10-30 22:13:19.810970153 +0000 UTC m=+1377.415277705"
Oct 30 22:13:20.039563 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:20.039490    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-zl688"
Oct 30 22:13:20.040928 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:20.040903    5678 patch_prober.go:29] interesting pod/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-zl688 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Oct 30 22:13:20.041055 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:20.040948    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-zl688" podUID="c6373802-bc1e-4606-b946-4ae9753c560a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Oct 30 22:13:20.581782 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:20.578546    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-zl688" event={"ID":"c6373802-bc1e-4606-b946-4ae9753c560a","Type":"ContainerStarted","Data":"fad35cf21cdba6b83a349e4852266f7563805393e18a06784f6b76505d00090d"}
Oct 30 22:13:20.758427 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:20.757786    5678 patch_prober.go:29] interesting pod/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-q828p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Oct 30 22:13:20.758427 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:20.757849    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-q828p" podUID="4e8c9108-98ce-4a09-a8a9-f13def9228df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Oct 30 22:13:21.061071 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:21.061033    5678 patch_prober.go:29] interesting pod/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-zl688 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Oct 30 22:13:21.061696 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:21.061662    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-zl688" podUID="c6373802-bc1e-4606-b946-4ae9753c560a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Oct 30 22:13:21.756426 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:21.750536    5678 patch_prober.go:29] interesting pod/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-q828p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Oct 30 22:13:21.756426 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:21.750604    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-q828p" podUID="4e8c9108-98ce-4a09-a8a9-f13def9228df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Oct 30 22:13:22.067480 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:22.066800    5678 patch_prober.go:29] interesting pod/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-zl688 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Oct 30 22:13:22.067480 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:22.066858    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-zl688" podUID="c6373802-bc1e-4606-b946-4ae9753c560a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Oct 30 22:13:22.745423 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:22.745225    5678 patch_prober.go:29] interesting pod/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-q828p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Oct 30 22:13:22.745423 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:22.745299    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-q828p" podUID="4e8c9108-98ce-4a09-a8a9-f13def9228df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Oct 30 22:13:23.042475 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:23.040784    5678 patch_prober.go:29] interesting pod/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-zl688 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Oct 30 22:13:23.042475 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:23.040842    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-zl688" podUID="c6373802-bc1e-4606-b946-4ae9753c560a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Oct 30 22:13:23.761425 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:23.751795    5678 patch_prober.go:29] interesting pod/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-q828p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Oct 30 22:13:23.761425 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:23.751865    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-q828p" podUID="4e8c9108-98ce-4a09-a8a9-f13def9228df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Oct 30 22:13:24.040475 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:24.039795    5678 patch_prober.go:29] interesting pod/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-zl688 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Oct 30 22:13:24.040475 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:24.039870    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-zl688" podUID="c6373802-bc1e-4606-b946-4ae9753c560a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Oct 30 22:13:24.749426 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:24.745853    5678 patch_prober.go:29] interesting pod/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-q828p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Oct 30 22:13:24.749426 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:24.745940    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-q828p" podUID="4e8c9108-98ce-4a09-a8a9-f13def9228df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Oct 30 22:13:25.049486 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:25.049080    5678 patch_prober.go:29] interesting pod/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-zl688 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Oct 30 22:13:25.049486 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:25.049154    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-zl688" podUID="c6373802-bc1e-4606-b946-4ae9753c560a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Oct 30 22:13:25.751426 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:25.750822    5678 patch_prober.go:29] interesting pod/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-q828p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Oct 30 22:13:25.751426 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:25.750892    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-q828p" podUID="4e8c9108-98ce-4a09-a8a9-f13def9228df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Oct 30 22:13:26.042477 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:26.041808    5678 patch_prober.go:29] interesting pod/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-zl688 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Oct 30 22:13:26.042477 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:26.041867    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-zl688" podUID="c6373802-bc1e-4606-b946-4ae9753c560a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Oct 30 22:13:26.200250 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:26.200216    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:13:26.200687 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:26.200283    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:13:26.719534 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:26.719489    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9bfdx" event={"ID":"1b614bbd-09b5-4b79-a9a5-1c57ae795852","Type":"ContainerStarted","Data":"8afe92c9b204030cfad6f3dc612d17be23f6d48382a88b78b930024f42bd1235"}
Oct 30 22:13:26.720171 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:26.720141    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9bfdx"
Oct 30 22:13:26.745289 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:26.745251    5678 patch_prober.go:29] interesting pod/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-q828p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Oct 30 22:13:26.745486 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:26.745314    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-q828p" podUID="4e8c9108-98ce-4a09-a8a9-f13def9228df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Oct 30 22:13:26.838552 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:26.838498    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9bfdx" podStartSLOduration=3.029964473 podStartE2EDuration="9.838480985s" totalImagesPullingTime="6.808516512s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:13:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:13:26.836795659 +0000 UTC m=+1384.441103237" watchObservedRunningTime="2026-10-30 22:13:26.838480985 +0000 UTC m=+1384.442788539"
Oct 30 22:13:27.040189 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:27.040100    5678 patch_prober.go:29] interesting pod/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-zl688 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Oct 30 22:13:27.040350 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:27.040185    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-zl688" podUID="c6373802-bc1e-4606-b946-4ae9753c560a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Oct 30 22:13:27.736270 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:27.736233    5678 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-9bfdx container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:13:27.736776 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:27.736293    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9bfdx" podUID="1b614bbd-09b5-4b79-a9a5-1c57ae795852" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:13:27.750618 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:27.750592    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-q828p"
Oct 30 22:13:27.750949 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:27.750921    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-q828p"
Oct 30 22:13:27.751729 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:27.751711    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-q828p"
Oct 30 22:13:28.046485 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:28.046365    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-zl688"
Oct 30 22:13:28.047043 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:28.047012    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-zl688"
Oct 30 22:13:28.047989 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:28.047960    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-zl688"
Oct 30 22:13:29.072230 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:29.072193    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33af867b-fmbw4"]
Oct 30 22:13:29.116428 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:29.116382    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33af867b-fmbw4"]
Oct 30 22:13:29.120514 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:29.120495    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26472ab3-gr8x4"]
Oct 30 22:13:29.124882 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:29.124863    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26472ab3-gr8x4"
Oct 30 22:13:29.167806 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:29.167768    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26472ab3-gr8x4"]
Oct 30 22:13:29.175134 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:29.175107    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:13:29.175372 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:29.175356    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5rvcj\""
Oct 30 22:13:29.183846 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:29.183823    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5cd4402-8612-4915-a5bd-debb68d321be-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-26472ab3-gr8x4\" (UID: \"a5cd4402-8612-4915-a5bd-debb68d321be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26472ab3-gr8x4"
Oct 30 22:13:29.184002 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:29.183983    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5cd4402-8612-4915-a5bd-debb68d321be-maas-api-xks-ca\") pod \"maas-api-revoke-sub-26472ab3-gr8x4\" (UID: \"a5cd4402-8612-4915-a5bd-debb68d321be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26472ab3-gr8x4"
Oct 30 22:13:29.184080 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:29.184048    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xnbs\" (UniqueName: \"kubernetes.io/projected/a5cd4402-8612-4915-a5bd-debb68d321be-kube-api-access-8xnbs\") pod \"maas-api-revoke-sub-26472ab3-gr8x4\" (UID: \"a5cd4402-8612-4915-a5bd-debb68d321be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26472ab3-gr8x4"
Oct 30 22:13:29.285252 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:29.285220    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5cd4402-8612-4915-a5bd-debb68d321be-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-26472ab3-gr8x4\" (UID: \"a5cd4402-8612-4915-a5bd-debb68d321be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26472ab3-gr8x4"
Oct 30 22:13:29.285389 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:29.285312    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5cd4402-8612-4915-a5bd-debb68d321be-maas-api-xks-ca\") pod \"maas-api-revoke-sub-26472ab3-gr8x4\" (UID: \"a5cd4402-8612-4915-a5bd-debb68d321be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26472ab3-gr8x4"
Oct 30 22:13:29.285389 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:29.285347    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xnbs\" (UniqueName: \"kubernetes.io/projected/a5cd4402-8612-4915-a5bd-debb68d321be-kube-api-access-8xnbs\") pod \"maas-api-revoke-sub-26472ab3-gr8x4\" (UID: \"a5cd4402-8612-4915-a5bd-debb68d321be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26472ab3-gr8x4"
Oct 30 22:13:29.285913 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:29.285895    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5cd4402-8612-4915-a5bd-debb68d321be-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-26472ab3-gr8x4\" (UID: \"a5cd4402-8612-4915-a5bd-debb68d321be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26472ab3-gr8x4"
Oct 30 22:13:29.287637 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:29.287614    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5cd4402-8612-4915-a5bd-debb68d321be-maas-api-xks-ca\") pod \"maas-api-revoke-sub-26472ab3-gr8x4\" (UID: \"a5cd4402-8612-4915-a5bd-debb68d321be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26472ab3-gr8x4"
Oct 30 22:13:29.344252 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:29.344192    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xnbs\" (UniqueName: \"kubernetes.io/projected/a5cd4402-8612-4915-a5bd-debb68d321be-kube-api-access-8xnbs\") pod \"maas-api-revoke-sub-26472ab3-gr8x4\" (UID: \"a5cd4402-8612-4915-a5bd-debb68d321be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26472ab3-gr8x4"
Oct 30 22:13:29.438410 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:29.438378    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26472ab3-gr8x4"
Oct 30 22:13:29.826002 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:29.825970    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26472ab3-gr8x4"]
Oct 30 22:13:30.530244 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:30.530155    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e093ed7d-278e-45d1-8018-b7db3da90d17" path="/var/lib/kubelet/pods/e093ed7d-278e-45d1-8018-b7db3da90d17/volumes"
Oct 30 22:13:30.735690 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:30.735657    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9bfdx"
Oct 30 22:13:30.762704 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:30.762663    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26472ab3-gr8x4" event={"ID":"a5cd4402-8612-4915-a5bd-debb68d321be","Type":"ContainerStarted","Data":"6718a3fd2b71806234a6062c4f2b8773b433b276151b2989256274d61c519c9b"}
Oct 30 22:13:30.762881 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:30.762729    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26472ab3-gr8x4" event={"ID":"a5cd4402-8612-4915-a5bd-debb68d321be","Type":"ContainerStarted","Data":"5f064b314c126345e59a63cfd5f7dea1c99ff24ecd127da681c930b1ec62c8a8"}
Oct 30 22:13:31.018769 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:31.018718    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26472ab3-gr8x4" podStartSLOduration=2.018703843 podStartE2EDuration="2.018703843s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:13:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:13:31.01529401 +0000 UTC m=+1388.619601563" watchObservedRunningTime="2026-10-30 22:13:31.018703843 +0000 UTC m=+1388.623011396"
Oct 30 22:13:31.774071 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:31.774034    5678 generic.go:356] "Generic (PLEG): container finished" podID="a5cd4402-8612-4915-a5bd-debb68d321be" containerID="6718a3fd2b71806234a6062c4f2b8773b433b276151b2989256274d61c519c9b" exitCode=0
Oct 30 22:13:31.774631 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:31.774103    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26472ab3-gr8x4" event={"ID":"a5cd4402-8612-4915-a5bd-debb68d321be","Type":"ContainerDied","Data":"6718a3fd2b71806234a6062c4f2b8773b433b276151b2989256274d61c519c9b"}
Oct 30 22:13:33.667343 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:33.639339    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9ndd9" podUID="81bd6415-f42f-446b-a64e-956f794b3956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": context deadline exceeded"
Oct 30 22:13:33.692686 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:33.692607    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7bb1c2/premium-w3-7bb1c2-kserve-7dbf875d88-dqxn5" podUID="a58c911f-e2eb-4820-a6b6-24afe78b5fd4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:13:34.049434 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:34.041042    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb0b93/simulator-w6-fb0b93-kserve-55b784b479-588bn" podUID="951a6f7e-a22e-40f2-ace2-1ce5d3bc9b49" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:13:34.049434 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:34.047163    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-65e59f/simulator-w4-65e59f-kserve-85698bcd47-przch" podUID="ee5e843b-17be-4623-8c95-dc07c8b7f506" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Oct 30 22:13:34.099424 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:34.091584    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7bb1c2/simulator-w3-7bb1c2-kserve-675645869c-vcwqv" podUID="1fce9daf-68c9-41d0-825b-2012bb93c8ba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:13:34.113677 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:34.113637    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-65e59f/premium-w4-65e59f-kserve-6dc49468f8-rlzhx" podUID="8590a783-eb24-4759-ad74-1369f36aa9d2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:13:34.132440 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:34.120495    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-65e59f/distinct-2-w4-65e59f-kserve-689566757d-bf98q" podUID="64e66fdb-056e-4086-a85a-8c02613584a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": context deadline exceeded"
Oct 30 22:13:34.132966 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:34.132933    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779f32/premium-w0-779f32-kserve-67978797c8-jpkvl" podUID="40138fb9-9a75-4458-a351-11eb61c4a1e3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:13:34.150625 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:34.150595    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26472ab3-gr8x4"
Oct 30 22:13:34.392056 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:34.391999    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26472ab3-gr8x4" event={"ID":"a5cd4402-8612-4915-a5bd-debb68d321be","Type":"ContainerDied","Data":"5f064b314c126345e59a63cfd5f7dea1c99ff24ecd127da681c930b1ec62c8a8"}
Oct 30 22:13:34.421833 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:34.421806    5678 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5f064b314c126345e59a63cfd5f7dea1c99ff24ecd127da681c930b1ec62c8a8"
Oct 30 22:13:34.422056 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:34.407955    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5cd4402-8612-4915-a5bd-debb68d321be-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5cd4402-8612-4915-a5bd-debb68d321be-maas-api-openshift-ca\") pod \"a5cd4402-8612-4915-a5bd-debb68d321be\" (UID: \"a5cd4402-8612-4915-a5bd-debb68d321be\") "
Oct 30 22:13:34.422331 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:34.422310    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5cd4402-8612-4915-a5bd-debb68d321be-kube-api-access-8xnbs\" (UniqueName: \"kubernetes.io/projected/a5cd4402-8612-4915-a5bd-debb68d321be-kube-api-access-8xnbs\") pod \"a5cd4402-8612-4915-a5bd-debb68d321be\" (UID: \"a5cd4402-8612-4915-a5bd-debb68d321be\") "
Oct 30 22:13:34.430862 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:34.430837    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5cd4402-8612-4915-a5bd-debb68d321be-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5cd4402-8612-4915-a5bd-debb68d321be-maas-api-xks-ca\") pod \"a5cd4402-8612-4915-a5bd-debb68d321be\" (UID: \"a5cd4402-8612-4915-a5bd-debb68d321be\") "
Oct 30 22:13:34.436516 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:34.406131    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26472ab3-gr8x4"
Oct 30 22:13:34.437105 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:34.437077    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5cd4402-8612-4915-a5bd-debb68d321be-maas-api-openshift-ca" pod "a5cd4402-8612-4915-a5bd-debb68d321be" (UID: "a5cd4402-8612-4915-a5bd-debb68d321be"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:13:34.630298 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:34.630261    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5cd4402-8612-4915-a5bd-debb68d321be-maas-api-openshift-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:13:34.689704 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:34.689620    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9ndd9" podUID="81bd6415-f42f-446b-a64e-956f794b3956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:13:34.733902 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:34.733861    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5cd4402-8612-4915-a5bd-debb68d321be-maas-api-xks-ca" pod "a5cd4402-8612-4915-a5bd-debb68d321be" (UID: "a5cd4402-8612-4915-a5bd-debb68d321be"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:13:34.735168 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:34.735120    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5cd4402-8612-4915-a5bd-debb68d321be-kube-api-access-8xnbs" pod "a5cd4402-8612-4915-a5bd-debb68d321be" (UID: "a5cd4402-8612-4915-a5bd-debb68d321be"). InnerVolumeSpecName "kube-api-access-8xnbs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:13:34.735946 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:34.735798    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9ndd9"
Oct 30 22:13:34.744341 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:34.744306    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9ndd9"
Oct 30 22:13:34.834865 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:34.834824    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xnbs\" (UniqueName: \"kubernetes.io/projected/a5cd4402-8612-4915-a5bd-debb68d321be-kube-api-access-8xnbs\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:13:34.834865 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:34.834859    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5cd4402-8612-4915-a5bd-debb68d321be-maas-api-xks-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:13:37.266022 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:37.265986    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2395b65-2ngnr"]
Oct 30 22:13:37.266735 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:37.266717    5678 cpu_manager.go:414] "containerMap: removing container" podUID="a5cd4402-8612-4915-a5bd-debb68d321be" containerName="revoke-keys"
Oct 30 22:13:37.266781 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:37.266737    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5cd4402-8612-4915-a5bd-debb68d321be" containerName="revoke-keys"
Oct 30 22:13:37.266854 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:37.266846    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5cd4402-8612-4915-a5bd-debb68d321be" containerName="revoke-keys"
Oct 30 22:13:37.270651 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:37.270634    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2395b65-2ngnr"
Oct 30 22:13:37.326225 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:37.326193    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2395b65-2ngnr"]
Oct 30 22:13:37.367431 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:37.367373    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:13:37.367635 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:37.367563    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5rvcj\""
Oct 30 22:13:37.372643 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:37.372610    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c4f4212-b490-49a0-9e31-f74e31ec0f2f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2395b65-2ngnr\" (UID: \"9c4f4212-b490-49a0-9e31-f74e31ec0f2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2395b65-2ngnr"
Oct 30 22:13:37.372900 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:37.372875    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76cp7\" (UniqueName: \"kubernetes.io/projected/9c4f4212-b490-49a0-9e31-f74e31ec0f2f-kube-api-access-76cp7\") pod \"maas-api-revoke-sub-e2395b65-2ngnr\" (UID: \"9c4f4212-b490-49a0-9e31-f74e31ec0f2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2395b65-2ngnr"
Oct 30 22:13:37.372979 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:37.372966    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c4f4212-b490-49a0-9e31-f74e31ec0f2f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2395b65-2ngnr\" (UID: \"9c4f4212-b490-49a0-9e31-f74e31ec0f2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2395b65-2ngnr"
Oct 30 22:13:37.474710 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:37.474672    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c4f4212-b490-49a0-9e31-f74e31ec0f2f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2395b65-2ngnr\" (UID: \"9c4f4212-b490-49a0-9e31-f74e31ec0f2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2395b65-2ngnr"
Oct 30 22:13:37.474880 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:37.474835    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76cp7\" (UniqueName: \"kubernetes.io/projected/9c4f4212-b490-49a0-9e31-f74e31ec0f2f-kube-api-access-76cp7\") pod \"maas-api-revoke-sub-e2395b65-2ngnr\" (UID: \"9c4f4212-b490-49a0-9e31-f74e31ec0f2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2395b65-2ngnr"
Oct 30 22:13:37.474880 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:37.474872    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c4f4212-b490-49a0-9e31-f74e31ec0f2f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2395b65-2ngnr\" (UID: \"9c4f4212-b490-49a0-9e31-f74e31ec0f2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2395b65-2ngnr"
Oct 30 22:13:37.475454 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:37.475432    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c4f4212-b490-49a0-9e31-f74e31ec0f2f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2395b65-2ngnr\" (UID: \"9c4f4212-b490-49a0-9e31-f74e31ec0f2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2395b65-2ngnr"
Oct 30 22:13:37.477024 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:37.477000    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c4f4212-b490-49a0-9e31-f74e31ec0f2f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2395b65-2ngnr\" (UID: \"9c4f4212-b490-49a0-9e31-f74e31ec0f2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2395b65-2ngnr"
Oct 30 22:13:37.533737 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:37.533658    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76cp7\" (UniqueName: \"kubernetes.io/projected/9c4f4212-b490-49a0-9e31-f74e31ec0f2f-kube-api-access-76cp7\") pod \"maas-api-revoke-sub-e2395b65-2ngnr\" (UID: \"9c4f4212-b490-49a0-9e31-f74e31ec0f2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2395b65-2ngnr"
Oct 30 22:13:37.583954 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:37.583923    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2395b65-2ngnr"
Oct 30 22:13:37.989719 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:13:37.989686    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c4f4212_b490_49a0_9e31_f74e31ec0f2f.slice/crio-004404249c5d166f4ad2de1c42b5e77706472197f4121e52dbf48a9baaeb2034 WatchSource:0}: Error finding container 004404249c5d166f4ad2de1c42b5e77706472197f4121e52dbf48a9baaeb2034: Status 404 returned error can't find the container with id 004404249c5d166f4ad2de1c42b5e77706472197f4121e52dbf48a9baaeb2034
Oct 30 22:13:37.990429 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:37.990387    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2395b65-2ngnr"]
Oct 30 22:13:38.427186 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:38.426896    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2395b65-2ngnr" event={"ID":"9c4f4212-b490-49a0-9e31-f74e31ec0f2f","Type":"ContainerStarted","Data":"9a25a0a5ed3ed0a9868be33935fc6b2a3f37167675aa545218d996929a1ad98e"}
Oct 30 22:13:38.427186 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:38.426974    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2395b65-2ngnr" event={"ID":"9c4f4212-b490-49a0-9e31-f74e31ec0f2f","Type":"ContainerStarted","Data":"004404249c5d166f4ad2de1c42b5e77706472197f4121e52dbf48a9baaeb2034"}
Oct 30 22:13:38.538831 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:38.538725    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2395b65-2ngnr" podStartSLOduration=1.538708625 podStartE2EDuration="1.538708625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:13:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:13:38.537091991 +0000 UTC m=+1396.141399555" watchObservedRunningTime="2026-10-30 22:13:38.538708625 +0000 UTC m=+1396.143016181"
Oct 30 22:13:39.053710 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:39.053679    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88b75b32-w7mh9"]
Oct 30 22:13:39.058134 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:39.058106    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88b75b32-w7mh9"]
Oct 30 22:13:39.436753 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:39.436719    5678 generic.go:356] "Generic (PLEG): container finished" podID="9c4f4212-b490-49a0-9e31-f74e31ec0f2f" containerID="9a25a0a5ed3ed0a9868be33935fc6b2a3f37167675aa545218d996929a1ad98e" exitCode=0
Oct 30 22:13:39.437167 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:39.436806    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2395b65-2ngnr" event={"ID":"9c4f4212-b490-49a0-9e31-f74e31ec0f2f","Type":"ContainerDied","Data":"9a25a0a5ed3ed0a9868be33935fc6b2a3f37167675aa545218d996929a1ad98e"}
Oct 30 22:13:40.525583 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:40.525549    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc690a1c-e106-4abd-b6ce-0ef6841374cc" path="/var/lib/kubelet/pods/bc690a1c-e106-4abd-b6ce-0ef6841374cc/volumes"
Oct 30 22:13:40.580957 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:13:40.580921    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 22:13:40.614740 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:40.614717    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2395b65-2ngnr"
Oct 30 22:13:40.728076 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:40.727967    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c4f4212-b490-49a0-9e31-f74e31ec0f2f-kube-api-access-76cp7\" (UniqueName: \"kubernetes.io/projected/9c4f4212-b490-49a0-9e31-f74e31ec0f2f-kube-api-access-76cp7\") pod \"9c4f4212-b490-49a0-9e31-f74e31ec0f2f\" (UID: \"9c4f4212-b490-49a0-9e31-f74e31ec0f2f\") "
Oct 30 22:13:40.728285 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:40.728231    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c4f4212-b490-49a0-9e31-f74e31ec0f2f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c4f4212-b490-49a0-9e31-f74e31ec0f2f-maas-api-openshift-ca\") pod \"9c4f4212-b490-49a0-9e31-f74e31ec0f2f\" (UID: \"9c4f4212-b490-49a0-9e31-f74e31ec0f2f\") "
Oct 30 22:13:40.728358 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:40.728288    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c4f4212-b490-49a0-9e31-f74e31ec0f2f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c4f4212-b490-49a0-9e31-f74e31ec0f2f-maas-api-xks-ca\") pod \"9c4f4212-b490-49a0-9e31-f74e31ec0f2f\" (UID: \"9c4f4212-b490-49a0-9e31-f74e31ec0f2f\") "
Oct 30 22:13:40.729615 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:40.729559    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c4f4212-b490-49a0-9e31-f74e31ec0f2f-maas-api-openshift-ca" pod "9c4f4212-b490-49a0-9e31-f74e31ec0f2f" (UID: "9c4f4212-b490-49a0-9e31-f74e31ec0f2f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:13:40.731586 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:40.731559    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c4f4212-b490-49a0-9e31-f74e31ec0f2f-maas-api-xks-ca" pod "9c4f4212-b490-49a0-9e31-f74e31ec0f2f" (UID: "9c4f4212-b490-49a0-9e31-f74e31ec0f2f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:13:40.732384 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:40.732364    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c4f4212-b490-49a0-9e31-f74e31ec0f2f-kube-api-access-76cp7" pod "9c4f4212-b490-49a0-9e31-f74e31ec0f2f" (UID: "9c4f4212-b490-49a0-9e31-f74e31ec0f2f"). InnerVolumeSpecName "kube-api-access-76cp7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:13:40.829859 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:40.829829    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76cp7\" (UniqueName: \"kubernetes.io/projected/9c4f4212-b490-49a0-9e31-f74e31ec0f2f-kube-api-access-76cp7\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:13:40.829859 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:40.829857    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c4f4212-b490-49a0-9e31-f74e31ec0f2f-maas-api-openshift-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:13:40.829859 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:40.829869    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c4f4212-b490-49a0-9e31-f74e31ec0f2f-maas-api-xks-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:13:41.452364 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:41.452336    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2395b65-2ngnr"
Oct 30 22:13:41.452556 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:41.452374    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2395b65-2ngnr" event={"ID":"9c4f4212-b490-49a0-9e31-f74e31ec0f2f","Type":"ContainerDied","Data":"004404249c5d166f4ad2de1c42b5e77706472197f4121e52dbf48a9baaeb2034"}
Oct 30 22:13:41.452556 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:41.452448    5678 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="004404249c5d166f4ad2de1c42b5e77706472197f4121e52dbf48a9baaeb2034"
Oct 30 22:13:44.050381 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:44.050345    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d5eb2f4-26gc5"]
Oct 30 22:13:44.099478 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:44.099447    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d5eb2f4-26gc5"]
Oct 30 22:13:44.526095 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:44.526062    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8394754e-3f5a-43e1-99df-1349985623f5" path="/var/lib/kubelet/pods/8394754e-3f5a-43e1-99df-1349985623f5/volumes"
Oct 30 22:13:47.660133 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:47.660100    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bwbzj"]
Oct 30 22:13:47.660828 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:47.660810    5678 cpu_manager.go:414] "containerMap: removing container" podUID="9c4f4212-b490-49a0-9e31-f74e31ec0f2f" containerName="revoke-keys"
Oct 30 22:13:47.660876 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:47.660830    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c4f4212-b490-49a0-9e31-f74e31ec0f2f" containerName="revoke-keys"
Oct 30 22:13:47.660957 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:47.660948    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c4f4212-b490-49a0-9e31-f74e31ec0f2f" containerName="revoke-keys"
Oct 30 22:13:47.664452 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:47.664430    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bwbzj"
Oct 30 22:13:47.723340 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:47.723305    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bwbzj"]
Oct 30 22:13:47.832144 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:47.832112    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62c63093-87f0-4e2d-81b7-5f8cc31cee82-credential-socket\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bwbzj\" (UID: \"62c63093-87f0-4e2d-81b7-5f8cc31cee82\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bwbzj"
Oct 30 22:13:47.832309 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:47.832151    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62c63093-87f0-4e2d-81b7-5f8cc31cee82-workload-socket\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bwbzj\" (UID: \"62c63093-87f0-4e2d-81b7-5f8cc31cee82\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bwbzj"
Oct 30 22:13:47.832309 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:47.832179    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62c63093-87f0-4e2d-81b7-5f8cc31cee82-workload-certs\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bwbzj\" (UID: \"62c63093-87f0-4e2d-81b7-5f8cc31cee82\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bwbzj"
Oct 30 22:13:47.832309 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:47.832286    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62c63093-87f0-4e2d-81b7-5f8cc31cee82-istio-data\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bwbzj\" (UID: \"62c63093-87f0-4e2d-81b7-5f8cc31cee82\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bwbzj"
Oct 30 22:13:47.832309 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:47.832305    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62c63093-87f0-4e2d-81b7-5f8cc31cee82-istio-envoy\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bwbzj\" (UID: \"62c63093-87f0-4e2d-81b7-5f8cc31cee82\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bwbzj"
Oct 30 22:13:47.832478 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:47.832409    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62c63093-87f0-4e2d-81b7-5f8cc31cee82-istio-podinfo\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bwbzj\" (UID: \"62c63093-87f0-4e2d-81b7-5f8cc31cee82\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bwbzj"
Oct 30 22:13:47.832528 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:47.832512    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62c63093-87f0-4e2d-81b7-5f8cc31cee82-istiod-ca-cert\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bwbzj\" (UID: \"62c63093-87f0-4e2d-81b7-5f8cc31cee82\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bwbzj"
Oct 30 22:13:47.832567 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:47.832541    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62c63093-87f0-4e2d-81b7-5f8cc31cee82-istio-token\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bwbzj\" (UID: \"62c63093-87f0-4e2d-81b7-5f8cc31cee82\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bwbzj"
Oct 30 22:13:47.832618 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:47.832606    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jhr9\" (UniqueName: \"kubernetes.io/projected/62c63093-87f0-4e2d-81b7-5f8cc31cee82-kube-api-access-7jhr9\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bwbzj\" (UID: \"62c63093-87f0-4e2d-81b7-5f8cc31cee82\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bwbzj"
Oct 30 22:13:47.934035 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:47.933962    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62c63093-87f0-4e2d-81b7-5f8cc31cee82-istio-data\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bwbzj\" (UID: \"62c63093-87f0-4e2d-81b7-5f8cc31cee82\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bwbzj"
Oct 30 22:13:47.934035 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:47.933995    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62c63093-87f0-4e2d-81b7-5f8cc31cee82-istio-envoy\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bwbzj\" (UID: \"62c63093-87f0-4e2d-81b7-5f8cc31cee82\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bwbzj"
Oct 30 22:13:47.934236 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:47.934051    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62c63093-87f0-4e2d-81b7-5f8cc31cee82-istio-podinfo\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bwbzj\" (UID: \"62c63093-87f0-4e2d-81b7-5f8cc31cee82\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bwbzj"
Oct 30 22:13:47.934236 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:47.934111    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62c63093-87f0-4e2d-81b7-5f8cc31cee82-istiod-ca-cert\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bwbzj\" (UID: \"62c63093-87f0-4e2d-81b7-5f8cc31cee82\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bwbzj"
Oct 30 22:13:47.934236 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:47.934139    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62c63093-87f0-4e2d-81b7-5f8cc31cee82-istio-token\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bwbzj\" (UID: \"62c63093-87f0-4e2d-81b7-5f8cc31cee82\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bwbzj"
Oct 30 22:13:47.934236 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:47.934222    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jhr9\" (UniqueName: \"kubernetes.io/projected/62c63093-87f0-4e2d-81b7-5f8cc31cee82-kube-api-access-7jhr9\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bwbzj\" (UID: \"62c63093-87f0-4e2d-81b7-5f8cc31cee82\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bwbzj"
Oct 30 22:13:47.934443 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:47.934348    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62c63093-87f0-4e2d-81b7-5f8cc31cee82-credential-socket\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bwbzj\" (UID: \"62c63093-87f0-4e2d-81b7-5f8cc31cee82\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bwbzj"
Oct 30 22:13:47.934443 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:47.934379    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62c63093-87f0-4e2d-81b7-5f8cc31cee82-workload-socket\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bwbzj\" (UID: \"62c63093-87f0-4e2d-81b7-5f8cc31cee82\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bwbzj"
Oct 30 22:13:47.934443 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:47.934431    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62c63093-87f0-4e2d-81b7-5f8cc31cee82-workload-certs\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bwbzj\" (UID: \"62c63093-87f0-4e2d-81b7-5f8cc31cee82\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bwbzj"
Oct 30 22:13:47.934607 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:47.934440    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62c63093-87f0-4e2d-81b7-5f8cc31cee82-istio-data\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bwbzj\" (UID: \"62c63093-87f0-4e2d-81b7-5f8cc31cee82\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bwbzj"
Oct 30 22:13:47.934798 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:47.934750    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62c63093-87f0-4e2d-81b7-5f8cc31cee82-workload-certs\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bwbzj\" (UID: \"62c63093-87f0-4e2d-81b7-5f8cc31cee82\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bwbzj"
Oct 30 22:13:47.934863 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:47.934841    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62c63093-87f0-4e2d-81b7-5f8cc31cee82-credential-socket\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bwbzj\" (UID: \"62c63093-87f0-4e2d-81b7-5f8cc31cee82\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bwbzj"
Oct 30 22:13:47.934930 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:47.934906    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62c63093-87f0-4e2d-81b7-5f8cc31cee82-workload-socket\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bwbzj\" (UID: \"62c63093-87f0-4e2d-81b7-5f8cc31cee82\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bwbzj"
Oct 30 22:13:47.934965 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:47.934914    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62c63093-87f0-4e2d-81b7-5f8cc31cee82-istiod-ca-cert\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bwbzj\" (UID: \"62c63093-87f0-4e2d-81b7-5f8cc31cee82\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bwbzj"
Oct 30 22:13:47.936453 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:47.936433    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62c63093-87f0-4e2d-81b7-5f8cc31cee82-istio-envoy\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bwbzj\" (UID: \"62c63093-87f0-4e2d-81b7-5f8cc31cee82\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bwbzj"
Oct 30 22:13:47.936526 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:47.936462    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62c63093-87f0-4e2d-81b7-5f8cc31cee82-istio-podinfo\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bwbzj\" (UID: \"62c63093-87f0-4e2d-81b7-5f8cc31cee82\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bwbzj"
Oct 30 22:13:48.006755 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:48.006725    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62c63093-87f0-4e2d-81b7-5f8cc31cee82-istio-token\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bwbzj\" (UID: \"62c63093-87f0-4e2d-81b7-5f8cc31cee82\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bwbzj"
Oct 30 22:13:48.009174 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:48.009140    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jhr9\" (UniqueName: \"kubernetes.io/projected/62c63093-87f0-4e2d-81b7-5f8cc31cee82-kube-api-access-7jhr9\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bwbzj\" (UID: \"62c63093-87f0-4e2d-81b7-5f8cc31cee82\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bwbzj"
Oct 30 22:13:48.057927 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:48.057898    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74aa8537-zxf2j"]
Oct 30 22:13:48.063786 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:48.063760    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74aa8537-zxf2j"]
Oct 30 22:13:48.278911 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:48.278835    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bwbzj"
Oct 30 22:13:48.524499 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:48.524469    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a617d026-0fd3-469b-87ec-47b0f4d85ad5" path="/var/lib/kubelet/pods/a617d026-0fd3-469b-87ec-47b0f4d85ad5/volumes"
Oct 30 22:13:48.719424 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:13:48.719381    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62c63093_87f0_4e2d_81b7_5f8cc31cee82.slice/crio-fd0e5ceb882a5175b2998655952f73bef91f8b610e071ce09bb0a4b8ae686805 WatchSource:0}: Error finding container fd0e5ceb882a5175b2998655952f73bef91f8b610e071ce09bb0a4b8ae686805: Status 404 returned error can't find the container with id fd0e5ceb882a5175b2998655952f73bef91f8b610e071ce09bb0a4b8ae686805
Oct 30 22:13:48.719870 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:48.719848    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bwbzj"]
Oct 30 22:13:48.722272 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:48.722217    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:13:48.722360 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:48.722303    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:13:48.722360 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:48.722333    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:13:49.518339 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:49.518302    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bwbzj" event={"ID":"62c63093-87f0-4e2d-81b7-5f8cc31cee82","Type":"ContainerStarted","Data":"047cf19b07e620417137aa117f6984b58ea0d7be0ad9e06f5bb6a795d04fc3c5"}
Oct 30 22:13:49.518536 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:49.518366    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bwbzj" event={"ID":"62c63093-87f0-4e2d-81b7-5f8cc31cee82","Type":"ContainerStarted","Data":"fd0e5ceb882a5175b2998655952f73bef91f8b610e071ce09bb0a4b8ae686805"}
Oct 30 22:13:49.684076 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:49.684010    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bwbzj" podStartSLOduration=2.683995855 podStartE2EDuration="2.683995855s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:13:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:13:49.680851185 +0000 UTC m=+1407.285158738" watchObservedRunningTime="2026-10-30 22:13:49.683995855 +0000 UTC m=+1407.288303408"
Oct 30 22:13:50.279619 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:50.279590    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bwbzj"
Oct 30 22:13:50.280926 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:50.280903    5678 patch_prober.go:29] interesting pod/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bwbzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Oct 30 22:13:50.281045 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:50.280955    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bwbzj" podUID="62c63093-87f0-4e2d-81b7-5f8cc31cee82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Oct 30 22:13:51.279838 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:51.279806    5678 patch_prober.go:29] interesting pod/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bwbzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Oct 30 22:13:51.280251 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:51.279868    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bwbzj" podUID="62c63093-87f0-4e2d-81b7-5f8cc31cee82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Oct 30 22:13:52.279179 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:52.279141    5678 patch_prober.go:29] interesting pod/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bwbzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Oct 30 22:13:52.279462 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:52.279198    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bwbzj" podUID="62c63093-87f0-4e2d-81b7-5f8cc31cee82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Oct 30 22:13:53.280125 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:53.280088    5678 patch_prober.go:29] interesting pod/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bwbzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Oct 30 22:13:53.280548 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:53.280143    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bwbzj" podUID="62c63093-87f0-4e2d-81b7-5f8cc31cee82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Oct 30 22:13:54.279950 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:54.279918    5678 patch_prober.go:29] interesting pod/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bwbzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Oct 30 22:13:54.280126 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:54.279968    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bwbzj" podUID="62c63093-87f0-4e2d-81b7-5f8cc31cee82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Oct 30 22:13:54.495794 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:54.495762    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a7082f2-b6vzv"]
Oct 30 22:13:54.501648 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:54.501628    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a7082f2-b6vzv"
Oct 30 22:13:54.558629 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:54.558542    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a7082f2-b6vzv"]
Oct 30 22:13:54.566029 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:54.566001    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwqwj\" (UniqueName: \"kubernetes.io/projected/3abb3e26-65da-49e8-85cf-7d0b01a1da7c-kube-api-access-mwqwj\") pod \"maas-api-revoke-sub-2a7082f2-b6vzv\" (UID: \"3abb3e26-65da-49e8-85cf-7d0b01a1da7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a7082f2-b6vzv"
Oct 30 22:13:54.566189 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:54.566044    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3abb3e26-65da-49e8-85cf-7d0b01a1da7c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a7082f2-b6vzv\" (UID: \"3abb3e26-65da-49e8-85cf-7d0b01a1da7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a7082f2-b6vzv"
Oct 30 22:13:54.566268 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:54.566243    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3abb3e26-65da-49e8-85cf-7d0b01a1da7c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a7082f2-b6vzv\" (UID: \"3abb3e26-65da-49e8-85cf-7d0b01a1da7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a7082f2-b6vzv"
Oct 30 22:13:54.597692 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:54.597664    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:13:54.597851 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:54.597778    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5rvcj\""
Oct 30 22:13:54.667585 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:54.667550    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3abb3e26-65da-49e8-85cf-7d0b01a1da7c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a7082f2-b6vzv\" (UID: \"3abb3e26-65da-49e8-85cf-7d0b01a1da7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a7082f2-b6vzv"
Oct 30 22:13:54.667789 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:54.667752    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwqwj\" (UniqueName: \"kubernetes.io/projected/3abb3e26-65da-49e8-85cf-7d0b01a1da7c-kube-api-access-mwqwj\") pod \"maas-api-revoke-sub-2a7082f2-b6vzv\" (UID: \"3abb3e26-65da-49e8-85cf-7d0b01a1da7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a7082f2-b6vzv"
Oct 30 22:13:54.667861 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:54.667812    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3abb3e26-65da-49e8-85cf-7d0b01a1da7c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a7082f2-b6vzv\" (UID: \"3abb3e26-65da-49e8-85cf-7d0b01a1da7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a7082f2-b6vzv"
Oct 30 22:13:54.668548 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:54.668526    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3abb3e26-65da-49e8-85cf-7d0b01a1da7c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a7082f2-b6vzv\" (UID: \"3abb3e26-65da-49e8-85cf-7d0b01a1da7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a7082f2-b6vzv"
Oct 30 22:13:54.670067 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:54.670038    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3abb3e26-65da-49e8-85cf-7d0b01a1da7c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a7082f2-b6vzv\" (UID: \"3abb3e26-65da-49e8-85cf-7d0b01a1da7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a7082f2-b6vzv"
Oct 30 22:13:54.748637 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:54.748607    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwqwj\" (UniqueName: \"kubernetes.io/projected/3abb3e26-65da-49e8-85cf-7d0b01a1da7c-kube-api-access-mwqwj\") pod \"maas-api-revoke-sub-2a7082f2-b6vzv\" (UID: \"3abb3e26-65da-49e8-85cf-7d0b01a1da7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a7082f2-b6vzv"
Oct 30 22:13:54.816953 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:54.816882    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a7082f2-b6vzv"
Oct 30 22:13:55.061080 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:55.061048    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e429ea7c-trt9p"]
Oct 30 22:13:55.068419 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:55.068339    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebb90d20-csn9b"]
Oct 30 22:13:55.106923 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:55.106860    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e429ea7c-trt9p"]
Oct 30 22:13:55.114857 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:55.114391    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebb90d20-csn9b"]
Oct 30 22:13:55.253485 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:55.253451    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a7082f2-b6vzv"]
Oct 30 22:13:55.279871 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:55.279846    5678 patch_prober.go:29] interesting pod/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bwbzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Oct 30 22:13:55.280006 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:55.279892    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bwbzj" podUID="62c63093-87f0-4e2d-81b7-5f8cc31cee82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Oct 30 22:13:55.598291 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:55.598258    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a7082f2-b6vzv" event={"ID":"3abb3e26-65da-49e8-85cf-7d0b01a1da7c","Type":"ContainerStarted","Data":"eea81a21b0822262309c1bf7bb93cd6f8789330db8428dc08f7d202ee687bb2f"}
Oct 30 22:13:55.598687 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:55.598310    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a7082f2-b6vzv" event={"ID":"3abb3e26-65da-49e8-85cf-7d0b01a1da7c","Type":"ContainerStarted","Data":"232107e8d665e4b153a6b02a51fe97fed39439b91538d090ce02dde5dd6e61be"}
Oct 30 22:13:55.709925 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:55.709833    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a7082f2-b6vzv" podStartSLOduration=1.709818305 podStartE2EDuration="1.709818305s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:13:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:13:55.709466766 +0000 UTC m=+1413.313774321" watchObservedRunningTime="2026-10-30 22:13:55.709818305 +0000 UTC m=+1413.314125861"
Oct 30 22:13:56.283962 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:56.283932    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bwbzj"
Oct 30 22:13:56.284454 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:56.284437    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bwbzj"
Oct 30 22:13:56.285155 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:56.285135    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bwbzj"
Oct 30 22:13:56.524694 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:56.524657    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d0dc96c-49eb-4b32-9154-d512fcef9f05" path="/var/lib/kubelet/pods/4d0dc96c-49eb-4b32-9154-d512fcef9f05/volumes"
Oct 30 22:13:56.525103 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:56.525088    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65e57873-3428-4f06-90f7-d8e0a49c0f9e" path="/var/lib/kubelet/pods/65e57873-3428-4f06-90f7-d8e0a49c0f9e/volumes"
Oct 30 22:13:56.609015 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:56.608982    5678 generic.go:356] "Generic (PLEG): container finished" podID="3abb3e26-65da-49e8-85cf-7d0b01a1da7c" containerID="eea81a21b0822262309c1bf7bb93cd6f8789330db8428dc08f7d202ee687bb2f" exitCode=0
Oct 30 22:13:56.609431 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:56.609065    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a7082f2-b6vzv" event={"ID":"3abb3e26-65da-49e8-85cf-7d0b01a1da7c","Type":"ContainerDied","Data":"eea81a21b0822262309c1bf7bb93cd6f8789330db8428dc08f7d202ee687bb2f"}
Oct 30 22:13:57.794185 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:57.794162    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a7082f2-b6vzv"
Oct 30 22:13:58.029518 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:58.029431    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3abb3e26-65da-49e8-85cf-7d0b01a1da7c-kube-api-access-mwqwj\" (UniqueName: \"kubernetes.io/projected/3abb3e26-65da-49e8-85cf-7d0b01a1da7c-kube-api-access-mwqwj\") pod \"3abb3e26-65da-49e8-85cf-7d0b01a1da7c\" (UID: \"3abb3e26-65da-49e8-85cf-7d0b01a1da7c\") "
Oct 30 22:13:58.029843 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:58.029809    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3abb3e26-65da-49e8-85cf-7d0b01a1da7c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3abb3e26-65da-49e8-85cf-7d0b01a1da7c-maas-api-xks-ca\") pod \"3abb3e26-65da-49e8-85cf-7d0b01a1da7c\" (UID: \"3abb3e26-65da-49e8-85cf-7d0b01a1da7c\") "
Oct 30 22:13:58.029983 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:58.029877    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3abb3e26-65da-49e8-85cf-7d0b01a1da7c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3abb3e26-65da-49e8-85cf-7d0b01a1da7c-maas-api-openshift-ca\") pod \"3abb3e26-65da-49e8-85cf-7d0b01a1da7c\" (UID: \"3abb3e26-65da-49e8-85cf-7d0b01a1da7c\") "
Oct 30 22:13:58.031107 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:58.031079    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3abb3e26-65da-49e8-85cf-7d0b01a1da7c-maas-api-openshift-ca" pod "3abb3e26-65da-49e8-85cf-7d0b01a1da7c" (UID: "3abb3e26-65da-49e8-85cf-7d0b01a1da7c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:13:58.033803 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:58.033778    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3abb3e26-65da-49e8-85cf-7d0b01a1da7c-kube-api-access-mwqwj" pod "3abb3e26-65da-49e8-85cf-7d0b01a1da7c" (UID: "3abb3e26-65da-49e8-85cf-7d0b01a1da7c"). InnerVolumeSpecName "kube-api-access-mwqwj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:13:58.069293 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:58.069144    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3abb3e26-65da-49e8-85cf-7d0b01a1da7c-maas-api-xks-ca" pod "3abb3e26-65da-49e8-85cf-7d0b01a1da7c" (UID: "3abb3e26-65da-49e8-85cf-7d0b01a1da7c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:13:58.132249 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:58.132206    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3abb3e26-65da-49e8-85cf-7d0b01a1da7c-maas-api-xks-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:13:58.132249 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:58.132252    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3abb3e26-65da-49e8-85cf-7d0b01a1da7c-maas-api-openshift-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:13:58.132473 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:58.132267    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwqwj\" (UniqueName: \"kubernetes.io/projected/3abb3e26-65da-49e8-85cf-7d0b01a1da7c-kube-api-access-mwqwj\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:13:58.663425 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:58.662917    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a7082f2-b6vzv" event={"ID":"3abb3e26-65da-49e8-85cf-7d0b01a1da7c","Type":"ContainerDied","Data":"232107e8d665e4b153a6b02a51fe97fed39439b91538d090ce02dde5dd6e61be"}
Oct 30 22:13:58.663425 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:58.663018    5678 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="232107e8d665e4b153a6b02a51fe97fed39439b91538d090ce02dde5dd6e61be"
Oct 30 22:13:58.663425 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:13:58.663091    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a7082f2-b6vzv"
Oct 30 22:14:01.460476 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:01.460443    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d74eab4-zvgf6"]
Oct 30 22:14:01.461198 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:01.461179    5678 cpu_manager.go:414] "containerMap: removing container" podUID="3abb3e26-65da-49e8-85cf-7d0b01a1da7c" containerName="revoke-keys"
Oct 30 22:14:01.461198 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:01.461198    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3abb3e26-65da-49e8-85cf-7d0b01a1da7c" containerName="revoke-keys"
Oct 30 22:14:01.461338 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:01.461323    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="3abb3e26-65da-49e8-85cf-7d0b01a1da7c" containerName="revoke-keys"
Oct 30 22:14:01.469357 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:01.469339    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d74eab4-zvgf6"
Oct 30 22:14:01.524195 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:01.524159    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d74eab4-zvgf6"]
Oct 30 22:14:01.563819 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:01.563788    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:14:01.564915 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:01.564869    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5rvcj\""
Oct 30 22:14:01.599153 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:01.599112    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwz2x\" (UniqueName: \"kubernetes.io/projected/3c0aa3a7-b85d-4476-8cc1-73160b38374c-kube-api-access-dwz2x\") pod \"maas-api-revoke-sub-5d74eab4-zvgf6\" (UID: \"3c0aa3a7-b85d-4476-8cc1-73160b38374c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d74eab4-zvgf6"
Oct 30 22:14:01.599342 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:01.599202    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c0aa3a7-b85d-4476-8cc1-73160b38374c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d74eab4-zvgf6\" (UID: \"3c0aa3a7-b85d-4476-8cc1-73160b38374c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d74eab4-zvgf6"
Oct 30 22:14:01.599342 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:01.599297    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c0aa3a7-b85d-4476-8cc1-73160b38374c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d74eab4-zvgf6\" (UID: \"3c0aa3a7-b85d-4476-8cc1-73160b38374c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d74eab4-zvgf6"
Oct 30 22:14:01.701181 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:01.701149    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwz2x\" (UniqueName: \"kubernetes.io/projected/3c0aa3a7-b85d-4476-8cc1-73160b38374c-kube-api-access-dwz2x\") pod \"maas-api-revoke-sub-5d74eab4-zvgf6\" (UID: \"3c0aa3a7-b85d-4476-8cc1-73160b38374c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d74eab4-zvgf6"
Oct 30 22:14:01.701446 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:01.701222    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c0aa3a7-b85d-4476-8cc1-73160b38374c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d74eab4-zvgf6\" (UID: \"3c0aa3a7-b85d-4476-8cc1-73160b38374c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d74eab4-zvgf6"
Oct 30 22:14:01.701446 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:01.701359    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c0aa3a7-b85d-4476-8cc1-73160b38374c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d74eab4-zvgf6\" (UID: \"3c0aa3a7-b85d-4476-8cc1-73160b38374c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d74eab4-zvgf6"
Oct 30 22:14:01.702173 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:01.702149    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c0aa3a7-b85d-4476-8cc1-73160b38374c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d74eab4-zvgf6\" (UID: \"3c0aa3a7-b85d-4476-8cc1-73160b38374c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d74eab4-zvgf6"
Oct 30 22:14:01.704279 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:01.704256    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c0aa3a7-b85d-4476-8cc1-73160b38374c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d74eab4-zvgf6\" (UID: \"3c0aa3a7-b85d-4476-8cc1-73160b38374c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d74eab4-zvgf6"
Oct 30 22:14:01.760580 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:01.760508    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwz2x\" (UniqueName: \"kubernetes.io/projected/3c0aa3a7-b85d-4476-8cc1-73160b38374c-kube-api-access-dwz2x\") pod \"maas-api-revoke-sub-5d74eab4-zvgf6\" (UID: \"3c0aa3a7-b85d-4476-8cc1-73160b38374c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d74eab4-zvgf6"
Oct 30 22:14:01.783596 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:01.783560    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d74eab4-zvgf6"
Oct 30 22:14:02.181979 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:02.181948    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d74eab4-zvgf6"]
Oct 30 22:14:02.182264 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:14:02.182238    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c0aa3a7_b85d_4476_8cc1_73160b38374c.slice/crio-8b985a9aa6233d5e424a563859c5757e02ed96bb15b721162ce18a4fd84699c6 WatchSource:0}: Error finding container 8b985a9aa6233d5e424a563859c5757e02ed96bb15b721162ce18a4fd84699c6: Status 404 returned error can't find the container with id 8b985a9aa6233d5e424a563859c5757e02ed96bb15b721162ce18a4fd84699c6
Oct 30 22:14:02.722472 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:02.722363    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d74eab4-zvgf6" event={"ID":"3c0aa3a7-b85d-4476-8cc1-73160b38374c","Type":"ContainerStarted","Data":"77a5e895a63aac5e6945085eed7300922f6d097ab82a18a3b4123eb52a737abd"}
Oct 30 22:14:02.722472 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:02.722460    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d74eab4-zvgf6" event={"ID":"3c0aa3a7-b85d-4476-8cc1-73160b38374c","Type":"ContainerStarted","Data":"8b985a9aa6233d5e424a563859c5757e02ed96bb15b721162ce18a4fd84699c6"}
Oct 30 22:14:02.837396 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:02.837322    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d74eab4-zvgf6" podStartSLOduration=1.8373036520000001 podStartE2EDuration="1.837303652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:14:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:14:02.834113314 +0000 UTC m=+1420.438420868" watchObservedRunningTime="2026-10-30 22:14:02.837303652 +0000 UTC m=+1420.441611206"
Oct 30 22:14:03.732263 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:03.732227    5678 generic.go:356] "Generic (PLEG): container finished" podID="3c0aa3a7-b85d-4476-8cc1-73160b38374c" containerID="77a5e895a63aac5e6945085eed7300922f6d097ab82a18a3b4123eb52a737abd" exitCode=0
Oct 30 22:14:03.732669 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:03.732280    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d74eab4-zvgf6" event={"ID":"3c0aa3a7-b85d-4476-8cc1-73160b38374c","Type":"ContainerDied","Data":"77a5e895a63aac5e6945085eed7300922f6d097ab82a18a3b4123eb52a737abd"}
Oct 30 22:14:04.908186 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:04.908154    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d74eab4-zvgf6"
Oct 30 22:14:05.050720 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:05.050645    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c0aa3a7-b85d-4476-8cc1-73160b38374c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c0aa3a7-b85d-4476-8cc1-73160b38374c-maas-api-openshift-ca\") pod \"3c0aa3a7-b85d-4476-8cc1-73160b38374c\" (UID: \"3c0aa3a7-b85d-4476-8cc1-73160b38374c\") "
Oct 30 22:14:05.050899 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:05.050876    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c0aa3a7-b85d-4476-8cc1-73160b38374c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c0aa3a7-b85d-4476-8cc1-73160b38374c-maas-api-xks-ca\") pod \"3c0aa3a7-b85d-4476-8cc1-73160b38374c\" (UID: \"3c0aa3a7-b85d-4476-8cc1-73160b38374c\") "
Oct 30 22:14:05.051039 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:05.051022    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c0aa3a7-b85d-4476-8cc1-73160b38374c-kube-api-access-dwz2x\" (UniqueName: \"kubernetes.io/projected/3c0aa3a7-b85d-4476-8cc1-73160b38374c-kube-api-access-dwz2x\") pod \"3c0aa3a7-b85d-4476-8cc1-73160b38374c\" (UID: \"3c0aa3a7-b85d-4476-8cc1-73160b38374c\") "
Oct 30 22:14:05.051115 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:05.051050    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c0aa3a7-b85d-4476-8cc1-73160b38374c-maas-api-openshift-ca" pod "3c0aa3a7-b85d-4476-8cc1-73160b38374c" (UID: "3c0aa3a7-b85d-4476-8cc1-73160b38374c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:14:05.051824 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:05.051799    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c0aa3a7-b85d-4476-8cc1-73160b38374c-maas-api-openshift-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:14:05.053344 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:05.053322    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c0aa3a7-b85d-4476-8cc1-73160b38374c-maas-api-xks-ca" pod "3c0aa3a7-b85d-4476-8cc1-73160b38374c" (UID: "3c0aa3a7-b85d-4476-8cc1-73160b38374c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:14:05.053523 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:05.053498    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c0aa3a7-b85d-4476-8cc1-73160b38374c-kube-api-access-dwz2x" pod "3c0aa3a7-b85d-4476-8cc1-73160b38374c" (UID: "3c0aa3a7-b85d-4476-8cc1-73160b38374c"). InnerVolumeSpecName "kube-api-access-dwz2x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:14:05.153455 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:05.153387    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwz2x\" (UniqueName: \"kubernetes.io/projected/3c0aa3a7-b85d-4476-8cc1-73160b38374c-kube-api-access-dwz2x\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:14:05.153455 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:05.153455    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c0aa3a7-b85d-4476-8cc1-73160b38374c-maas-api-xks-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:14:05.749152 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:05.749120    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d74eab4-zvgf6"
Oct 30 22:14:05.749310 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:05.749147    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d74eab4-zvgf6" event={"ID":"3c0aa3a7-b85d-4476-8cc1-73160b38374c","Type":"ContainerDied","Data":"8b985a9aa6233d5e424a563859c5757e02ed96bb15b721162ce18a4fd84699c6"}
Oct 30 22:14:05.749310 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:05.749233    5678 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8b985a9aa6233d5e424a563859c5757e02ed96bb15b721162ce18a4fd84699c6"
Oct 30 22:14:06.060961 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:06.060867    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e648ccdf-5x4w5"]
Oct 30 22:14:06.065551 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:06.065524    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e648ccdf-5x4w5"]
Oct 30 22:14:06.524990 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:06.524958    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58553747-132d-4758-ad10-8df667f1c29d" path="/var/lib/kubelet/pods/58553747-132d-4758-ad10-8df667f1c29d/volumes"
Oct 30 22:14:07.043082 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:07.043053    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a05b2e8-shvt4"]
Oct 30 22:14:07.043681 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:07.043667    5678 cpu_manager.go:414] "containerMap: removing container" podUID="3c0aa3a7-b85d-4476-8cc1-73160b38374c" containerName="revoke-keys"
Oct 30 22:14:07.043729 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:07.043682    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c0aa3a7-b85d-4476-8cc1-73160b38374c" containerName="revoke-keys"
Oct 30 22:14:07.043812 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:07.043804    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c0aa3a7-b85d-4476-8cc1-73160b38374c" containerName="revoke-keys"
Oct 30 22:14:07.047708 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:07.047690    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a05b2e8-shvt4"
Oct 30 22:14:07.055881 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:07.055856    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e43442a0-9vt2m"]
Oct 30 22:14:07.077234 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:07.077211    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5152b232-a4a2-4b78-a6f9-f29695f54347-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a05b2e8-shvt4\" (UID: \"5152b232-a4a2-4b78-a6f9-f29695f54347\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a05b2e8-shvt4"
Oct 30 22:14:07.077552 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:07.077323    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8hb7\" (UniqueName: \"kubernetes.io/projected/5152b232-a4a2-4b78-a6f9-f29695f54347-kube-api-access-m8hb7\") pod \"maas-api-revoke-sub-7a05b2e8-shvt4\" (UID: \"5152b232-a4a2-4b78-a6f9-f29695f54347\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a05b2e8-shvt4"
Oct 30 22:14:07.077552 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:07.077529    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5152b232-a4a2-4b78-a6f9-f29695f54347-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a05b2e8-shvt4\" (UID: \"5152b232-a4a2-4b78-a6f9-f29695f54347\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a05b2e8-shvt4"
Oct 30 22:14:07.089723 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:07.089700    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e43442a0-9vt2m"]
Oct 30 22:14:07.097756 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:07.097735    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:14:07.101268 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:07.101249    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a05b2e8-shvt4"]
Oct 30 22:14:07.132704 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:07.132678    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5rvcj\""
Oct 30 22:14:07.179160 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:07.179125    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5152b232-a4a2-4b78-a6f9-f29695f54347-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a05b2e8-shvt4\" (UID: \"5152b232-a4a2-4b78-a6f9-f29695f54347\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a05b2e8-shvt4"
Oct 30 22:14:07.179314 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:07.179234    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8hb7\" (UniqueName: \"kubernetes.io/projected/5152b232-a4a2-4b78-a6f9-f29695f54347-kube-api-access-m8hb7\") pod \"maas-api-revoke-sub-7a05b2e8-shvt4\" (UID: \"5152b232-a4a2-4b78-a6f9-f29695f54347\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a05b2e8-shvt4"
Oct 30 22:14:07.179483 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:07.179464    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5152b232-a4a2-4b78-a6f9-f29695f54347-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a05b2e8-shvt4\" (UID: \"5152b232-a4a2-4b78-a6f9-f29695f54347\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a05b2e8-shvt4"
Oct 30 22:14:07.180192 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:07.180172    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5152b232-a4a2-4b78-a6f9-f29695f54347-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a05b2e8-shvt4\" (UID: \"5152b232-a4a2-4b78-a6f9-f29695f54347\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a05b2e8-shvt4"
Oct 30 22:14:07.182186 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:07.182164    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5152b232-a4a2-4b78-a6f9-f29695f54347-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a05b2e8-shvt4\" (UID: \"5152b232-a4a2-4b78-a6f9-f29695f54347\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a05b2e8-shvt4"
Oct 30 22:14:07.247552 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:07.247524    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8hb7\" (UniqueName: \"kubernetes.io/projected/5152b232-a4a2-4b78-a6f9-f29695f54347-kube-api-access-m8hb7\") pod \"maas-api-revoke-sub-7a05b2e8-shvt4\" (UID: \"5152b232-a4a2-4b78-a6f9-f29695f54347\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a05b2e8-shvt4"
Oct 30 22:14:07.362563 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:07.362479    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a05b2e8-shvt4"
Oct 30 22:14:07.759065 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:07.759035    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a05b2e8-shvt4"]
Oct 30 22:14:07.759198 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:14:07.759148    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5152b232_a4a2_4b78_a6f9_f29695f54347.slice/crio-02c420be205e5b16cd52cb4afbbf606e14a045cb7fb65104b8680c115577c2ab WatchSource:0}: Error finding container 02c420be205e5b16cd52cb4afbbf606e14a045cb7fb65104b8680c115577c2ab: Status 404 returned error can't find the container with id 02c420be205e5b16cd52cb4afbbf606e14a045cb7fb65104b8680c115577c2ab
Oct 30 22:14:07.771766 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:07.771734    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a05b2e8-shvt4" event={"ID":"5152b232-a4a2-4b78-a6f9-f29695f54347","Type":"ContainerStarted","Data":"02c420be205e5b16cd52cb4afbbf606e14a045cb7fb65104b8680c115577c2ab"}
Oct 30 22:14:08.529590 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:08.527752    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a6f4173-cdfa-434d-9a78-a433495e76ed" path="/var/lib/kubelet/pods/6a6f4173-cdfa-434d-9a78-a433495e76ed/volumes"
Oct 30 22:14:08.781888 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:08.781795    5678 generic.go:356] "Generic (PLEG): container finished" podID="5152b232-a4a2-4b78-a6f9-f29695f54347" containerID="4648bdf888daf6fc5d53e9b15701b7474718bebb0e66284398db0e9addff264e" exitCode=0
Oct 30 22:14:08.782039 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:08.781884    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a05b2e8-shvt4" event={"ID":"5152b232-a4a2-4b78-a6f9-f29695f54347","Type":"ContainerDied","Data":"4648bdf888daf6fc5d53e9b15701b7474718bebb0e66284398db0e9addff264e"}
Oct 30 22:14:09.962510 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:09.962487    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a05b2e8-shvt4"
Oct 30 22:14:10.018357 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:10.018327    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5152b232-a4a2-4b78-a6f9-f29695f54347-kube-api-access-m8hb7\" (UniqueName: \"kubernetes.io/projected/5152b232-a4a2-4b78-a6f9-f29695f54347-kube-api-access-m8hb7\") pod \"5152b232-a4a2-4b78-a6f9-f29695f54347\" (UID: \"5152b232-a4a2-4b78-a6f9-f29695f54347\") "
Oct 30 22:14:10.018642 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:10.018620    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5152b232-a4a2-4b78-a6f9-f29695f54347-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5152b232-a4a2-4b78-a6f9-f29695f54347-maas-api-xks-ca\") pod \"5152b232-a4a2-4b78-a6f9-f29695f54347\" (UID: \"5152b232-a4a2-4b78-a6f9-f29695f54347\") "
Oct 30 22:14:10.018793 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:10.018659    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5152b232-a4a2-4b78-a6f9-f29695f54347-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5152b232-a4a2-4b78-a6f9-f29695f54347-maas-api-openshift-ca\") pod \"5152b232-a4a2-4b78-a6f9-f29695f54347\" (UID: \"5152b232-a4a2-4b78-a6f9-f29695f54347\") "
Oct 30 22:14:10.019199 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:10.019171    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5152b232-a4a2-4b78-a6f9-f29695f54347-maas-api-openshift-ca" pod "5152b232-a4a2-4b78-a6f9-f29695f54347" (UID: "5152b232-a4a2-4b78-a6f9-f29695f54347"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:14:10.019337 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:10.019315    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5152b232-a4a2-4b78-a6f9-f29695f54347-maas-api-openshift-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:14:10.020531 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:10.020508    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5152b232-a4a2-4b78-a6f9-f29695f54347-kube-api-access-m8hb7" pod "5152b232-a4a2-4b78-a6f9-f29695f54347" (UID: "5152b232-a4a2-4b78-a6f9-f29695f54347"). InnerVolumeSpecName "kube-api-access-m8hb7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:14:10.020828 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:10.020807    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5152b232-a4a2-4b78-a6f9-f29695f54347-maas-api-xks-ca" pod "5152b232-a4a2-4b78-a6f9-f29695f54347" (UID: "5152b232-a4a2-4b78-a6f9-f29695f54347"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:14:10.120937 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:10.120904    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5152b232-a4a2-4b78-a6f9-f29695f54347-maas-api-xks-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:14:10.121081 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:10.120940    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8hb7\" (UniqueName: \"kubernetes.io/projected/5152b232-a4a2-4b78-a6f9-f29695f54347-kube-api-access-m8hb7\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:14:10.573188 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:14:10.573156    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 22:14:10.799034 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:10.798955    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a05b2e8-shvt4"
Oct 30 22:14:10.799034 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:10.798990    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a05b2e8-shvt4" event={"ID":"5152b232-a4a2-4b78-a6f9-f29695f54347","Type":"ContainerDied","Data":"02c420be205e5b16cd52cb4afbbf606e14a045cb7fb65104b8680c115577c2ab"}
Oct 30 22:14:10.799248 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:10.799037    5678 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="02c420be205e5b16cd52cb4afbbf606e14a045cb7fb65104b8680c115577c2ab"
Oct 30 22:14:11.052038 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:11.051954    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81e6c9d9-kbhlz"]
Oct 30 22:14:11.056192 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:11.056165    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81e6c9d9-kbhlz"]
Oct 30 22:14:12.079956 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:12.079924    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b531868-2xs6q"]
Oct 30 22:14:12.080617 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:12.080599    5678 cpu_manager.go:414] "containerMap: removing container" podUID="5152b232-a4a2-4b78-a6f9-f29695f54347" containerName="revoke-keys"
Oct 30 22:14:12.080617 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:12.080616    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5152b232-a4a2-4b78-a6f9-f29695f54347" containerName="revoke-keys"
Oct 30 22:14:12.080738 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:12.080720    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="5152b232-a4a2-4b78-a6f9-f29695f54347" containerName="revoke-keys"
Oct 30 22:14:12.084424 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:12.084394    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b531868-2xs6q"
Oct 30 22:14:12.138973 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:12.138943    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b531868-2xs6q"]
Oct 30 22:14:12.149247 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:12.149218    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/839a167c-bed2-434e-8a2c-31635f2c467b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b531868-2xs6q\" (UID: \"839a167c-bed2-434e-8a2c-31635f2c467b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b531868-2xs6q"
Oct 30 22:14:12.149432 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:12.149263    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v57xd\" (UniqueName: \"kubernetes.io/projected/839a167c-bed2-434e-8a2c-31635f2c467b-kube-api-access-v57xd\") pod \"maas-api-revoke-sub-8b531868-2xs6q\" (UID: \"839a167c-bed2-434e-8a2c-31635f2c467b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b531868-2xs6q"
Oct 30 22:14:12.149432 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:12.149295    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/839a167c-bed2-434e-8a2c-31635f2c467b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b531868-2xs6q\" (UID: \"839a167c-bed2-434e-8a2c-31635f2c467b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b531868-2xs6q"
Oct 30 22:14:12.180863 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:12.180826    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:14:12.181065 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:12.181041    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5rvcj\""
Oct 30 22:14:12.250635 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:12.250600    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/839a167c-bed2-434e-8a2c-31635f2c467b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b531868-2xs6q\" (UID: \"839a167c-bed2-434e-8a2c-31635f2c467b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b531868-2xs6q"
Oct 30 22:14:12.250796 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:12.250660    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v57xd\" (UniqueName: \"kubernetes.io/projected/839a167c-bed2-434e-8a2c-31635f2c467b-kube-api-access-v57xd\") pod \"maas-api-revoke-sub-8b531868-2xs6q\" (UID: \"839a167c-bed2-434e-8a2c-31635f2c467b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b531868-2xs6q"
Oct 30 22:14:12.250796 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:12.250697    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/839a167c-bed2-434e-8a2c-31635f2c467b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b531868-2xs6q\" (UID: \"839a167c-bed2-434e-8a2c-31635f2c467b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b531868-2xs6q"
Oct 30 22:14:12.251423 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:12.251387    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/839a167c-bed2-434e-8a2c-31635f2c467b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b531868-2xs6q\" (UID: \"839a167c-bed2-434e-8a2c-31635f2c467b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b531868-2xs6q"
Oct 30 22:14:12.253058 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:12.253035    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/839a167c-bed2-434e-8a2c-31635f2c467b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b531868-2xs6q\" (UID: \"839a167c-bed2-434e-8a2c-31635f2c467b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b531868-2xs6q"
Oct 30 22:14:12.312358 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:12.312333    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v57xd\" (UniqueName: \"kubernetes.io/projected/839a167c-bed2-434e-8a2c-31635f2c467b-kube-api-access-v57xd\") pod \"maas-api-revoke-sub-8b531868-2xs6q\" (UID: \"839a167c-bed2-434e-8a2c-31635f2c467b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b531868-2xs6q"
Oct 30 22:14:12.398123 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:12.398092    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b531868-2xs6q"
Oct 30 22:14:12.525780 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:12.525749    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8de2e4f8-e41f-466f-b376-6408c6ac5d65" path="/var/lib/kubelet/pods/8de2e4f8-e41f-466f-b376-6408c6ac5d65/volumes"
Oct 30 22:14:12.780296 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:14:12.780268    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod839a167c_bed2_434e_8a2c_31635f2c467b.slice/crio-fdd4354a6e860413ba5dc9cbe7bc9db11b49cc629177d6197a6b354819aafdba WatchSource:0}: Error finding container fdd4354a6e860413ba5dc9cbe7bc9db11b49cc629177d6197a6b354819aafdba: Status 404 returned error can't find the container with id fdd4354a6e860413ba5dc9cbe7bc9db11b49cc629177d6197a6b354819aafdba
Oct 30 22:14:12.781259 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:12.781230    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b531868-2xs6q"]
Oct 30 22:14:12.815450 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:12.815424    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b531868-2xs6q" event={"ID":"839a167c-bed2-434e-8a2c-31635f2c467b","Type":"ContainerStarted","Data":"fdd4354a6e860413ba5dc9cbe7bc9db11b49cc629177d6197a6b354819aafdba"}
Oct 30 22:14:13.825695 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:13.825659    5678 generic.go:356] "Generic (PLEG): container finished" podID="839a167c-bed2-434e-8a2c-31635f2c467b" containerID="159b56382d26b5e30f19e0c5d18735482fa9ebcc05d20397a7dd897521fbfdfa" exitCode=0
Oct 30 22:14:13.826047 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:13.825739    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b531868-2xs6q" event={"ID":"839a167c-bed2-434e-8a2c-31635f2c467b","Type":"ContainerDied","Data":"159b56382d26b5e30f19e0c5d18735482fa9ebcc05d20397a7dd897521fbfdfa"}
Oct 30 22:14:14.476063 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:14.476015    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-663bc197-khq9q"]
Oct 30 22:14:14.480447 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:14.480391    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-663bc197-khq9q"
Oct 30 22:14:14.536364 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:14.536334    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-663bc197-khq9q"]
Oct 30 22:14:14.585264 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:14.585232    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60cf0c3e-6cb6-4090-8831-0ca156c59337-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-663bc197-khq9q\" (UID: \"60cf0c3e-6cb6-4090-8831-0ca156c59337\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-663bc197-khq9q"
Oct 30 22:14:14.585438 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:14.585369    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxntw\" (UniqueName: \"kubernetes.io/projected/60cf0c3e-6cb6-4090-8831-0ca156c59337-kube-api-access-jxntw\") pod \"maas-api-revoke-sub-663bc197-khq9q\" (UID: \"60cf0c3e-6cb6-4090-8831-0ca156c59337\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-663bc197-khq9q"
Oct 30 22:14:14.585573 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:14.585557    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60cf0c3e-6cb6-4090-8831-0ca156c59337-maas-api-xks-ca\") pod \"maas-api-revoke-sub-663bc197-khq9q\" (UID: \"60cf0c3e-6cb6-4090-8831-0ca156c59337\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-663bc197-khq9q"
Oct 30 22:14:14.686594 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:14.686562    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxntw\" (UniqueName: \"kubernetes.io/projected/60cf0c3e-6cb6-4090-8831-0ca156c59337-kube-api-access-jxntw\") pod \"maas-api-revoke-sub-663bc197-khq9q\" (UID: \"60cf0c3e-6cb6-4090-8831-0ca156c59337\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-663bc197-khq9q"
Oct 30 22:14:14.686756 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:14.686693    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60cf0c3e-6cb6-4090-8831-0ca156c59337-maas-api-xks-ca\") pod \"maas-api-revoke-sub-663bc197-khq9q\" (UID: \"60cf0c3e-6cb6-4090-8831-0ca156c59337\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-663bc197-khq9q"
Oct 30 22:14:14.686857 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:14.686841    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60cf0c3e-6cb6-4090-8831-0ca156c59337-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-663bc197-khq9q\" (UID: \"60cf0c3e-6cb6-4090-8831-0ca156c59337\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-663bc197-khq9q"
Oct 30 22:14:14.687556 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:14.687529    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60cf0c3e-6cb6-4090-8831-0ca156c59337-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-663bc197-khq9q\" (UID: \"60cf0c3e-6cb6-4090-8831-0ca156c59337\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-663bc197-khq9q"
Oct 30 22:14:14.689451 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:14.689427    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60cf0c3e-6cb6-4090-8831-0ca156c59337-maas-api-xks-ca\") pod \"maas-api-revoke-sub-663bc197-khq9q\" (UID: \"60cf0c3e-6cb6-4090-8831-0ca156c59337\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-663bc197-khq9q"
Oct 30 22:14:14.745605 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:14.745548    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxntw\" (UniqueName: \"kubernetes.io/projected/60cf0c3e-6cb6-4090-8831-0ca156c59337-kube-api-access-jxntw\") pod \"maas-api-revoke-sub-663bc197-khq9q\" (UID: \"60cf0c3e-6cb6-4090-8831-0ca156c59337\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-663bc197-khq9q"
Oct 30 22:14:14.793678 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:14.793652    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-663bc197-khq9q"
Oct 30 22:14:15.021457 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:15.021383    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b531868-2xs6q"
Oct 30 22:14:15.093508 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:15.093440    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/839a167c-bed2-434e-8a2c-31635f2c467b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/839a167c-bed2-434e-8a2c-31635f2c467b-maas-api-xks-ca\") pod \"839a167c-bed2-434e-8a2c-31635f2c467b\" (UID: \"839a167c-bed2-434e-8a2c-31635f2c467b\") "
Oct 30 22:14:15.093508 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:15.093490    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/839a167c-bed2-434e-8a2c-31635f2c467b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/839a167c-bed2-434e-8a2c-31635f2c467b-maas-api-openshift-ca\") pod \"839a167c-bed2-434e-8a2c-31635f2c467b\" (UID: \"839a167c-bed2-434e-8a2c-31635f2c467b\") "
Oct 30 22:14:15.093705 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:15.093688    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/839a167c-bed2-434e-8a2c-31635f2c467b-kube-api-access-v57xd\" (UniqueName: \"kubernetes.io/projected/839a167c-bed2-434e-8a2c-31635f2c467b-kube-api-access-v57xd\") pod \"839a167c-bed2-434e-8a2c-31635f2c467b\" (UID: \"839a167c-bed2-434e-8a2c-31635f2c467b\") "
Oct 30 22:14:15.093902 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:15.093875    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/839a167c-bed2-434e-8a2c-31635f2c467b-maas-api-openshift-ca" pod "839a167c-bed2-434e-8a2c-31635f2c467b" (UID: "839a167c-bed2-434e-8a2c-31635f2c467b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:14:15.094242 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:15.094225    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/839a167c-bed2-434e-8a2c-31635f2c467b-maas-api-openshift-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:14:15.095387 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:15.095363    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/839a167c-bed2-434e-8a2c-31635f2c467b-maas-api-xks-ca" pod "839a167c-bed2-434e-8a2c-31635f2c467b" (UID: "839a167c-bed2-434e-8a2c-31635f2c467b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:14:15.095543 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:15.095526    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/839a167c-bed2-434e-8a2c-31635f2c467b-kube-api-access-v57xd" pod "839a167c-bed2-434e-8a2c-31635f2c467b" (UID: "839a167c-bed2-434e-8a2c-31635f2c467b"). InnerVolumeSpecName "kube-api-access-v57xd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:14:15.180941 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:15.180867    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-663bc197-khq9q"]
Oct 30 22:14:15.195601 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:15.195573    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/839a167c-bed2-434e-8a2c-31635f2c467b-maas-api-xks-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:14:15.195601 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:15.195604    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v57xd\" (UniqueName: \"kubernetes.io/projected/839a167c-bed2-434e-8a2c-31635f2c467b-kube-api-access-v57xd\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:14:15.219325 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:14:15.219280    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60cf0c3e_6cb6_4090_8831_0ca156c59337.slice/crio-4a78396e76b04b44f22537d81788db5588704e270fb6461ebbf230b906863450 WatchSource:0}: Error finding container 4a78396e76b04b44f22537d81788db5588704e270fb6461ebbf230b906863450: Status 404 returned error can't find the container with id 4a78396e76b04b44f22537d81788db5588704e270fb6461ebbf230b906863450
Oct 30 22:14:15.845366 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:15.845280    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b531868-2xs6q"
Oct 30 22:14:15.845555 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:15.845275    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b531868-2xs6q" event={"ID":"839a167c-bed2-434e-8a2c-31635f2c467b","Type":"ContainerDied","Data":"fdd4354a6e860413ba5dc9cbe7bc9db11b49cc629177d6197a6b354819aafdba"}
Oct 30 22:14:15.845555 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:15.845449    5678 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fdd4354a6e860413ba5dc9cbe7bc9db11b49cc629177d6197a6b354819aafdba"
Oct 30 22:14:15.847186 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:15.847162    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-663bc197-khq9q" event={"ID":"60cf0c3e-6cb6-4090-8831-0ca156c59337","Type":"ContainerStarted","Data":"3646900a47069009182de0f1460707bc1eac9d55cf690cd9925e764a6162c31f"}
Oct 30 22:14:15.847327 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:15.847202    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-663bc197-khq9q" event={"ID":"60cf0c3e-6cb6-4090-8831-0ca156c59337","Type":"ContainerStarted","Data":"4a78396e76b04b44f22537d81788db5588704e270fb6461ebbf230b906863450"}
Oct 30 22:14:15.965772 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:15.965722    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-663bc197-khq9q" podStartSLOduration=1.965708325 podStartE2EDuration="1.965708325s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:14:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:14:15.961287939 +0000 UTC m=+1433.565595491" watchObservedRunningTime="2026-10-30 22:14:15.965708325 +0000 UTC m=+1433.570015878"
Oct 30 22:14:16.856997 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:16.856956    5678 generic.go:356] "Generic (PLEG): container finished" podID="60cf0c3e-6cb6-4090-8831-0ca156c59337" containerID="3646900a47069009182de0f1460707bc1eac9d55cf690cd9925e764a6162c31f" exitCode=0
Oct 30 22:14:16.857378 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:16.857035    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-663bc197-khq9q" event={"ID":"60cf0c3e-6cb6-4090-8831-0ca156c59337","Type":"ContainerDied","Data":"3646900a47069009182de0f1460707bc1eac9d55cf690cd9925e764a6162c31f"}
Oct 30 22:14:17.054150 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:17.054117    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00d9b33b-tcsgs"]
Oct 30 22:14:17.059065 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:17.059039    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00d9b33b-tcsgs"]
Oct 30 22:14:18.029506 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:18.029483    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-663bc197-khq9q"
Oct 30 22:14:18.145475 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:18.145361    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60cf0c3e-6cb6-4090-8831-0ca156c59337-kube-api-access-jxntw\" (UniqueName: \"kubernetes.io/projected/60cf0c3e-6cb6-4090-8831-0ca156c59337-kube-api-access-jxntw\") pod \"60cf0c3e-6cb6-4090-8831-0ca156c59337\" (UID: \"60cf0c3e-6cb6-4090-8831-0ca156c59337\") "
Oct 30 22:14:18.145475 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:18.145449    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60cf0c3e-6cb6-4090-8831-0ca156c59337-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60cf0c3e-6cb6-4090-8831-0ca156c59337-maas-api-openshift-ca\") pod \"60cf0c3e-6cb6-4090-8831-0ca156c59337\" (UID: \"60cf0c3e-6cb6-4090-8831-0ca156c59337\") "
Oct 30 22:14:18.145676 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:18.145636    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60cf0c3e-6cb6-4090-8831-0ca156c59337-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60cf0c3e-6cb6-4090-8831-0ca156c59337-maas-api-xks-ca\") pod \"60cf0c3e-6cb6-4090-8831-0ca156c59337\" (UID: \"60cf0c3e-6cb6-4090-8831-0ca156c59337\") "
Oct 30 22:14:18.146690 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:18.146664    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60cf0c3e-6cb6-4090-8831-0ca156c59337-maas-api-openshift-ca" pod "60cf0c3e-6cb6-4090-8831-0ca156c59337" (UID: "60cf0c3e-6cb6-4090-8831-0ca156c59337"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:14:18.148338 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:18.148312    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60cf0c3e-6cb6-4090-8831-0ca156c59337-maas-api-xks-ca" pod "60cf0c3e-6cb6-4090-8831-0ca156c59337" (UID: "60cf0c3e-6cb6-4090-8831-0ca156c59337"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:14:18.149105 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:18.149040    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60cf0c3e-6cb6-4090-8831-0ca156c59337-kube-api-access-jxntw" pod "60cf0c3e-6cb6-4090-8831-0ca156c59337" (UID: "60cf0c3e-6cb6-4090-8831-0ca156c59337"). InnerVolumeSpecName "kube-api-access-jxntw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:14:18.247871 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:18.247834    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60cf0c3e-6cb6-4090-8831-0ca156c59337-maas-api-xks-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:14:18.247871 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:18.247866    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxntw\" (UniqueName: \"kubernetes.io/projected/60cf0c3e-6cb6-4090-8831-0ca156c59337-kube-api-access-jxntw\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:14:18.248075 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:18.247884    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60cf0c3e-6cb6-4090-8831-0ca156c59337-maas-api-openshift-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:14:18.526350 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:18.526278    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61aa9c47-8147-4d4e-852d-0973ba1c5c01" path="/var/lib/kubelet/pods/61aa9c47-8147-4d4e-852d-0973ba1c5c01/volumes"
Oct 30 22:14:18.875655 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:18.875627    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-663bc197-khq9q"
Oct 30 22:14:18.875810 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:18.875651    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-663bc197-khq9q" event={"ID":"60cf0c3e-6cb6-4090-8831-0ca156c59337","Type":"ContainerDied","Data":"4a78396e76b04b44f22537d81788db5588704e270fb6461ebbf230b906863450"}
Oct 30 22:14:18.875810 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:18.875709    5678 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4a78396e76b04b44f22537d81788db5588704e270fb6461ebbf230b906863450"
Oct 30 22:14:22.371124 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:22.371072    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-58b3d74d-g984m"]
Oct 30 22:14:22.371961 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:22.371938    5678 cpu_manager.go:414] "containerMap: removing container" podUID="839a167c-bed2-434e-8a2c-31635f2c467b" containerName="revoke-keys"
Oct 30 22:14:22.372038 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:22.371963    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="839a167c-bed2-434e-8a2c-31635f2c467b" containerName="revoke-keys"
Oct 30 22:14:22.372038 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:22.371977    5678 cpu_manager.go:414] "containerMap: removing container" podUID="60cf0c3e-6cb6-4090-8831-0ca156c59337" containerName="revoke-keys"
Oct 30 22:14:22.372038 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:22.371983    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60cf0c3e-6cb6-4090-8831-0ca156c59337" containerName="revoke-keys"
Oct 30 22:14:22.372152 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:22.372123    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="839a167c-bed2-434e-8a2c-31635f2c467b" containerName="revoke-keys"
Oct 30 22:14:22.372152 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:22.372136    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="60cf0c3e-6cb6-4090-8831-0ca156c59337" containerName="revoke-keys"
Oct 30 22:14:22.377079 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:22.377056    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58b3d74d-g984m"
Oct 30 22:14:22.429534 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:22.429504    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-58b3d74d-g984m"]
Oct 30 22:14:22.471388 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:22.471356    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:14:22.471569 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:22.471524    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5rvcj\""
Oct 30 22:14:22.507704 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:22.507674    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9q2h\" (UniqueName: \"kubernetes.io/projected/638dfd81-12d9-438a-83d9-8b42fd7d5d0f-kube-api-access-j9q2h\") pod \"maas-api-revoke-sub-58b3d74d-g984m\" (UID: \"638dfd81-12d9-438a-83d9-8b42fd7d5d0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58b3d74d-g984m"
Oct 30 22:14:22.507841 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:22.507716    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/638dfd81-12d9-438a-83d9-8b42fd7d5d0f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-58b3d74d-g984m\" (UID: \"638dfd81-12d9-438a-83d9-8b42fd7d5d0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58b3d74d-g984m"
Oct 30 22:14:22.507841 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:22.507736    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/638dfd81-12d9-438a-83d9-8b42fd7d5d0f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-58b3d74d-g984m\" (UID: \"638dfd81-12d9-438a-83d9-8b42fd7d5d0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58b3d74d-g984m"
Oct 30 22:14:22.609588 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:22.609551    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9q2h\" (UniqueName: \"kubernetes.io/projected/638dfd81-12d9-438a-83d9-8b42fd7d5d0f-kube-api-access-j9q2h\") pod \"maas-api-revoke-sub-58b3d74d-g984m\" (UID: \"638dfd81-12d9-438a-83d9-8b42fd7d5d0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58b3d74d-g984m"
Oct 30 22:14:22.609745 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:22.609679    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/638dfd81-12d9-438a-83d9-8b42fd7d5d0f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-58b3d74d-g984m\" (UID: \"638dfd81-12d9-438a-83d9-8b42fd7d5d0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58b3d74d-g984m"
Oct 30 22:14:22.609745 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:22.609715    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/638dfd81-12d9-438a-83d9-8b42fd7d5d0f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-58b3d74d-g984m\" (UID: \"638dfd81-12d9-438a-83d9-8b42fd7d5d0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58b3d74d-g984m"
Oct 30 22:14:22.610342 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:22.610323    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/638dfd81-12d9-438a-83d9-8b42fd7d5d0f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-58b3d74d-g984m\" (UID: \"638dfd81-12d9-438a-83d9-8b42fd7d5d0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58b3d74d-g984m"
Oct 30 22:14:22.658393 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:22.658329    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:14:22.662534 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:22.662512    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/638dfd81-12d9-438a-83d9-8b42fd7d5d0f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-58b3d74d-g984m\" (UID: \"638dfd81-12d9-438a-83d9-8b42fd7d5d0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58b3d74d-g984m"
Oct 30 22:14:22.667991 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:22.667974    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9q2h\" (UniqueName: \"kubernetes.io/projected/638dfd81-12d9-438a-83d9-8b42fd7d5d0f-kube-api-access-j9q2h\") pod \"maas-api-revoke-sub-58b3d74d-g984m\" (UID: \"638dfd81-12d9-438a-83d9-8b42fd7d5d0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58b3d74d-g984m"
Oct 30 22:14:22.714472 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:22.714444    5678 scope.go:122] "RemoveContainer" containerID="9983dc645593397b1963b2f6901b553d1c6083868a8550871624f89e29c535af"
Oct 30 22:14:22.741081 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:22.741061    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5rvcj\""
Oct 30 22:14:22.743169 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:22.743152    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58b3d74d-g984m"
Oct 30 22:14:23.134607 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:14:23.134574    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod638dfd81_12d9_438a_83d9_8b42fd7d5d0f.slice/crio-5ebb46251bae0a4e246df290ffb2286d84c2313dbd469f24a24190eabda69f3d WatchSource:0}: Error finding container 5ebb46251bae0a4e246df290ffb2286d84c2313dbd469f24a24190eabda69f3d: Status 404 returned error can't find the container with id 5ebb46251bae0a4e246df290ffb2286d84c2313dbd469f24a24190eabda69f3d
Oct 30 22:14:23.135943 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:23.135918    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-58b3d74d-g984m"]
Oct 30 22:14:23.918183 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:23.918146    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58b3d74d-g984m" event={"ID":"638dfd81-12d9-438a-83d9-8b42fd7d5d0f","Type":"ContainerStarted","Data":"8307e52fa9d117041c02f3569f43909a040daa38fa411cd9f4cb4b648c66a629"}
Oct 30 22:14:23.918725 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:23.918206    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58b3d74d-g984m" event={"ID":"638dfd81-12d9-438a-83d9-8b42fd7d5d0f","Type":"ContainerStarted","Data":"5ebb46251bae0a4e246df290ffb2286d84c2313dbd469f24a24190eabda69f3d"}
Oct 30 22:14:24.033811 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:24.033750    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58b3d74d-g984m" podStartSLOduration=2.033732322 podStartE2EDuration="2.033732322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:14:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:14:24.029487645 +0000 UTC m=+1441.633795198" watchObservedRunningTime="2026-10-30 22:14:24.033732322 +0000 UTC m=+1441.638039877"
Oct 30 22:14:24.062824 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:24.062791    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-344071f5-wnlx2"]
Oct 30 22:14:24.080942 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:24.080914    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-344071f5-wnlx2"]
Oct 30 22:14:24.524160 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:24.524127    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5191ae0-55f2-4180-b01e-226183a6a225" path="/var/lib/kubelet/pods/c5191ae0-55f2-4180-b01e-226183a6a225/volumes"
Oct 30 22:14:24.875626 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:24.875595    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3491e9e-jpg5w"]
Oct 30 22:14:24.880321 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:24.880301    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3491e9e-jpg5w"
Oct 30 22:14:24.927788 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:24.927742    5678 generic.go:356] "Generic (PLEG): container finished" podID="638dfd81-12d9-438a-83d9-8b42fd7d5d0f" containerID="8307e52fa9d117041c02f3569f43909a040daa38fa411cd9f4cb4b648c66a629" exitCode=0
Oct 30 22:14:24.928487 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:24.927792    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58b3d74d-g984m" event={"ID":"638dfd81-12d9-438a-83d9-8b42fd7d5d0f","Type":"ContainerDied","Data":"8307e52fa9d117041c02f3569f43909a040daa38fa411cd9f4cb4b648c66a629"}
Oct 30 22:14:24.934614 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:24.934588    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3491e9e-jpg5w"]
Oct 30 22:14:24.945158 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:24.945133    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9db99c04-1a54-413b-868b-196c86d4c074-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f3491e9e-jpg5w\" (UID: \"9db99c04-1a54-413b-868b-196c86d4c074\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3491e9e-jpg5w"
Oct 30 22:14:24.945308 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:24.945287    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9db99c04-1a54-413b-868b-196c86d4c074-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f3491e9e-jpg5w\" (UID: \"9db99c04-1a54-413b-868b-196c86d4c074\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3491e9e-jpg5w"
Oct 30 22:14:24.945360 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:24.945335    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snrlh\" (UniqueName: \"kubernetes.io/projected/9db99c04-1a54-413b-868b-196c86d4c074-kube-api-access-snrlh\") pod \"maas-api-revoke-sub-f3491e9e-jpg5w\" (UID: \"9db99c04-1a54-413b-868b-196c86d4c074\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3491e9e-jpg5w"
Oct 30 22:14:25.047800 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:25.047761    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9db99c04-1a54-413b-868b-196c86d4c074-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f3491e9e-jpg5w\" (UID: \"9db99c04-1a54-413b-868b-196c86d4c074\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3491e9e-jpg5w"
Oct 30 22:14:25.048807 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:25.048778    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snrlh\" (UniqueName: \"kubernetes.io/projected/9db99c04-1a54-413b-868b-196c86d4c074-kube-api-access-snrlh\") pod \"maas-api-revoke-sub-f3491e9e-jpg5w\" (UID: \"9db99c04-1a54-413b-868b-196c86d4c074\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3491e9e-jpg5w"
Oct 30 22:14:25.051912 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:25.051884    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9db99c04-1a54-413b-868b-196c86d4c074-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f3491e9e-jpg5w\" (UID: \"9db99c04-1a54-413b-868b-196c86d4c074\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3491e9e-jpg5w"
Oct 30 22:14:25.052291 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:25.052037    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9db99c04-1a54-413b-868b-196c86d4c074-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f3491e9e-jpg5w\" (UID: \"9db99c04-1a54-413b-868b-196c86d4c074\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3491e9e-jpg5w"
Oct 30 22:14:25.060600 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:25.060569    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9db99c04-1a54-413b-868b-196c86d4c074-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f3491e9e-jpg5w\" (UID: \"9db99c04-1a54-413b-868b-196c86d4c074\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3491e9e-jpg5w"
Oct 30 22:14:25.114293 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:25.114265    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-snrlh\" (UniqueName: \"kubernetes.io/projected/9db99c04-1a54-413b-868b-196c86d4c074-kube-api-access-snrlh\") pod \"maas-api-revoke-sub-f3491e9e-jpg5w\" (UID: \"9db99c04-1a54-413b-868b-196c86d4c074\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3491e9e-jpg5w"
Oct 30 22:14:25.194362 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:25.194277    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3491e9e-jpg5w"
Oct 30 22:14:25.598273 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:25.598244    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3491e9e-jpg5w"]
Oct 30 22:14:25.599615 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:14:25.599588    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9db99c04_1a54_413b_868b_196c86d4c074.slice/crio-c18dc65aa5758d339aa862e0cd3fef0e8ac1b54f917ca3be6a061c63b47d1d7d WatchSource:0}: Error finding container c18dc65aa5758d339aa862e0cd3fef0e8ac1b54f917ca3be6a061c63b47d1d7d: Status 404 returned error can't find the container with id c18dc65aa5758d339aa862e0cd3fef0e8ac1b54f917ca3be6a061c63b47d1d7d
Oct 30 22:14:25.939462 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:25.939429    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3491e9e-jpg5w" event={"ID":"9db99c04-1a54-413b-868b-196c86d4c074","Type":"ContainerStarted","Data":"c18dc65aa5758d339aa862e0cd3fef0e8ac1b54f917ca3be6a061c63b47d1d7d"}
Oct 30 22:14:26.115102 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:26.115075    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58b3d74d-g984m"
Oct 30 22:14:26.168384 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:26.168354    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/638dfd81-12d9-438a-83d9-8b42fd7d5d0f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/638dfd81-12d9-438a-83d9-8b42fd7d5d0f-maas-api-xks-ca\") pod \"638dfd81-12d9-438a-83d9-8b42fd7d5d0f\" (UID: \"638dfd81-12d9-438a-83d9-8b42fd7d5d0f\") "
Oct 30 22:14:26.168543 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:26.168430    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/638dfd81-12d9-438a-83d9-8b42fd7d5d0f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/638dfd81-12d9-438a-83d9-8b42fd7d5d0f-maas-api-openshift-ca\") pod \"638dfd81-12d9-438a-83d9-8b42fd7d5d0f\" (UID: \"638dfd81-12d9-438a-83d9-8b42fd7d5d0f\") "
Oct 30 22:14:26.168543 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:26.168498    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/638dfd81-12d9-438a-83d9-8b42fd7d5d0f-kube-api-access-j9q2h\" (UniqueName: \"kubernetes.io/projected/638dfd81-12d9-438a-83d9-8b42fd7d5d0f-kube-api-access-j9q2h\") pod \"638dfd81-12d9-438a-83d9-8b42fd7d5d0f\" (UID: \"638dfd81-12d9-438a-83d9-8b42fd7d5d0f\") "
Oct 30 22:14:26.168796 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:26.168770    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/638dfd81-12d9-438a-83d9-8b42fd7d5d0f-maas-api-openshift-ca" pod "638dfd81-12d9-438a-83d9-8b42fd7d5d0f" (UID: "638dfd81-12d9-438a-83d9-8b42fd7d5d0f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:14:26.169197 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:26.169175    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/638dfd81-12d9-438a-83d9-8b42fd7d5d0f-maas-api-openshift-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:14:26.170665 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:26.170636    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/638dfd81-12d9-438a-83d9-8b42fd7d5d0f-kube-api-access-j9q2h" pod "638dfd81-12d9-438a-83d9-8b42fd7d5d0f" (UID: "638dfd81-12d9-438a-83d9-8b42fd7d5d0f"). InnerVolumeSpecName "kube-api-access-j9q2h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:14:26.170921 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:26.170901    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/638dfd81-12d9-438a-83d9-8b42fd7d5d0f-maas-api-xks-ca" pod "638dfd81-12d9-438a-83d9-8b42fd7d5d0f" (UID: "638dfd81-12d9-438a-83d9-8b42fd7d5d0f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:14:26.270462 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:26.270381    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/638dfd81-12d9-438a-83d9-8b42fd7d5d0f-maas-api-xks-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:14:26.270462 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:26.270451    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9q2h\" (UniqueName: \"kubernetes.io/projected/638dfd81-12d9-438a-83d9-8b42fd7d5d0f-kube-api-access-j9q2h\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:14:26.948052 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:26.948015    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58b3d74d-g984m"
Oct 30 22:14:26.948535 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:26.948055    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58b3d74d-g984m" event={"ID":"638dfd81-12d9-438a-83d9-8b42fd7d5d0f","Type":"ContainerDied","Data":"5ebb46251bae0a4e246df290ffb2286d84c2313dbd469f24a24190eabda69f3d"}
Oct 30 22:14:26.948535 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:26.948113    5678 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5ebb46251bae0a4e246df290ffb2286d84c2313dbd469f24a24190eabda69f3d"
Oct 30 22:14:26.950024 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:26.949995    5678 generic.go:356] "Generic (PLEG): container finished" podID="9db99c04-1a54-413b-868b-196c86d4c074" containerID="b7f33affc7b6076c3dded4f89c2804eca32fc64323012fcc55cd108e879a23ac" exitCode=0
Oct 30 22:14:26.950168 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:26.950080    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3491e9e-jpg5w" event={"ID":"9db99c04-1a54-413b-868b-196c86d4c074","Type":"ContainerDied","Data":"b7f33affc7b6076c3dded4f89c2804eca32fc64323012fcc55cd108e879a23ac"}
Oct 30 22:14:27.348248 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:27.348162    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-9vxbk"]
Oct 30 22:14:27.348462 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:27.348436    5678 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-9vxbk" podUID="bde7f877-4938-42d4-be6c-cba04795ce83" containerName="istio-proxy" containerID="cri-o://026096dcf8dc7c8e0a89f9b9e951f419a9891095ba4a068e9cffe23d108aa87e" gracePeriod=30
Oct 30 22:14:28.127038 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:28.127015    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3491e9e-jpg5w"
Oct 30 22:14:28.199286 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:28.199258    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9db99c04-1a54-413b-868b-196c86d4c074-kube-api-access-snrlh\" (UniqueName: \"kubernetes.io/projected/9db99c04-1a54-413b-868b-196c86d4c074-kube-api-access-snrlh\") pod \"9db99c04-1a54-413b-868b-196c86d4c074\" (UID: \"9db99c04-1a54-413b-868b-196c86d4c074\") "
Oct 30 22:14:28.199476 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:28.199384    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9db99c04-1a54-413b-868b-196c86d4c074-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9db99c04-1a54-413b-868b-196c86d4c074-maas-api-openshift-ca\") pod \"9db99c04-1a54-413b-868b-196c86d4c074\" (UID: \"9db99c04-1a54-413b-868b-196c86d4c074\") "
Oct 30 22:14:28.199598 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:28.199584    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9db99c04-1a54-413b-868b-196c86d4c074-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9db99c04-1a54-413b-868b-196c86d4c074-maas-api-xks-ca\") pod \"9db99c04-1a54-413b-868b-196c86d4c074\" (UID: \"9db99c04-1a54-413b-868b-196c86d4c074\") "
Oct 30 22:14:28.199757 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:28.199730    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9db99c04-1a54-413b-868b-196c86d4c074-maas-api-openshift-ca" pod "9db99c04-1a54-413b-868b-196c86d4c074" (UID: "9db99c04-1a54-413b-868b-196c86d4c074"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:14:28.200067 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:28.200044    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9db99c04-1a54-413b-868b-196c86d4c074-maas-api-openshift-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:14:28.201597 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:28.201575    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9db99c04-1a54-413b-868b-196c86d4c074-maas-api-xks-ca" pod "9db99c04-1a54-413b-868b-196c86d4c074" (UID: "9db99c04-1a54-413b-868b-196c86d4c074"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:14:28.201705 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:28.201688    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9db99c04-1a54-413b-868b-196c86d4c074-kube-api-access-snrlh" pod "9db99c04-1a54-413b-868b-196c86d4c074" (UID: "9db99c04-1a54-413b-868b-196c86d4c074"). InnerVolumeSpecName "kube-api-access-snrlh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:14:28.301719 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:28.301684    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-snrlh\" (UniqueName: \"kubernetes.io/projected/9db99c04-1a54-413b-868b-196c86d4c074-kube-api-access-snrlh\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:14:28.301719 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:28.301717    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9db99c04-1a54-413b-868b-196c86d4c074-maas-api-xks-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:14:28.968095 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:28.968066    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3491e9e-jpg5w"
Oct 30 22:14:28.968277 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:28.968109    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3491e9e-jpg5w" event={"ID":"9db99c04-1a54-413b-868b-196c86d4c074","Type":"ContainerDied","Data":"c18dc65aa5758d339aa862e0cd3fef0e8ac1b54f917ca3be6a061c63b47d1d7d"}
Oct 30 22:14:28.968277 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:28.968201    5678 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c18dc65aa5758d339aa862e0cd3fef0e8ac1b54f917ca3be6a061c63b47d1d7d"
Oct 30 22:14:30.779832 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:30.779794    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3acbb058-68zvq"]
Oct 30 22:14:30.784167 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:30.780829    5678 cpu_manager.go:414] "containerMap: removing container" podUID="9db99c04-1a54-413b-868b-196c86d4c074" containerName="revoke-keys"
Oct 30 22:14:30.784167 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:30.780850    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9db99c04-1a54-413b-868b-196c86d4c074" containerName="revoke-keys"
Oct 30 22:14:30.784167 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:30.780875    5678 cpu_manager.go:414] "containerMap: removing container" podUID="638dfd81-12d9-438a-83d9-8b42fd7d5d0f" containerName="revoke-keys"
Oct 30 22:14:30.784167 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:30.780883    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="638dfd81-12d9-438a-83d9-8b42fd7d5d0f" containerName="revoke-keys"
Oct 30 22:14:30.784167 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:30.781026    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="9db99c04-1a54-413b-868b-196c86d4c074" containerName="revoke-keys"
Oct 30 22:14:30.784167 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:30.781041    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="638dfd81-12d9-438a-83d9-8b42fd7d5d0f" containerName="revoke-keys"
Oct 30 22:14:30.794021 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:30.793985    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3acbb058-68zvq"
Oct 30 22:14:30.825426 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:30.823994    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3acbb058-68zvq"]
Oct 30 22:14:30.852293 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:30.850508    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g84x2\" (UniqueName: \"kubernetes.io/projected/acfe6f2a-8f4e-4a7f-bde4-9486ed55cf07-kube-api-access-g84x2\") pod \"maas-api-revoke-sub-3acbb058-68zvq\" (UID: \"acfe6f2a-8f4e-4a7f-bde4-9486ed55cf07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3acbb058-68zvq"
Oct 30 22:14:30.852293 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:30.850603    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acfe6f2a-8f4e-4a7f-bde4-9486ed55cf07-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3acbb058-68zvq\" (UID: \"acfe6f2a-8f4e-4a7f-bde4-9486ed55cf07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3acbb058-68zvq"
Oct 30 22:14:30.852293 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:30.850787    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acfe6f2a-8f4e-4a7f-bde4-9486ed55cf07-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3acbb058-68zvq\" (UID: \"acfe6f2a-8f4e-4a7f-bde4-9486ed55cf07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3acbb058-68zvq"
Oct 30 22:14:30.906494 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:30.902683    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5rvcj\""
Oct 30 22:14:30.906494 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:30.902740    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:14:30.963461 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:30.952353    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g84x2\" (UniqueName: \"kubernetes.io/projected/acfe6f2a-8f4e-4a7f-bde4-9486ed55cf07-kube-api-access-g84x2\") pod \"maas-api-revoke-sub-3acbb058-68zvq\" (UID: \"acfe6f2a-8f4e-4a7f-bde4-9486ed55cf07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3acbb058-68zvq"
Oct 30 22:14:30.963461 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:30.952452    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acfe6f2a-8f4e-4a7f-bde4-9486ed55cf07-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3acbb058-68zvq\" (UID: \"acfe6f2a-8f4e-4a7f-bde4-9486ed55cf07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3acbb058-68zvq"
Oct 30 22:14:30.963461 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:30.952630    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acfe6f2a-8f4e-4a7f-bde4-9486ed55cf07-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3acbb058-68zvq\" (UID: \"acfe6f2a-8f4e-4a7f-bde4-9486ed55cf07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3acbb058-68zvq"
Oct 30 22:14:30.963461 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:30.956648    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acfe6f2a-8f4e-4a7f-bde4-9486ed55cf07-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3acbb058-68zvq\" (UID: \"acfe6f2a-8f4e-4a7f-bde4-9486ed55cf07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3acbb058-68zvq"
Oct 30 22:14:30.963461 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:30.962844    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acfe6f2a-8f4e-4a7f-bde4-9486ed55cf07-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3acbb058-68zvq\" (UID: \"acfe6f2a-8f4e-4a7f-bde4-9486ed55cf07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3acbb058-68zvq"
Oct 30 22:14:31.032469 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:31.030206    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g84x2\" (UniqueName: \"kubernetes.io/projected/acfe6f2a-8f4e-4a7f-bde4-9486ed55cf07-kube-api-access-g84x2\") pod \"maas-api-revoke-sub-3acbb058-68zvq\" (UID: \"acfe6f2a-8f4e-4a7f-bde4-9486ed55cf07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3acbb058-68zvq"
Oct 30 22:14:31.169481 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:31.161146    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3acbb058-68zvq"
Oct 30 22:14:32.633436 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:32.632049    5678 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-7bb1c2/distinct-w3-7bb1c2-kserve-8d695596c-8zgtw" podUID="36f0a6a2-54be-49bf-9202-502960cbb134" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": context deadline exceeded"
Oct 30 22:14:32.645475 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:32.637934    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7bb1c2/premium-w3-7bb1c2-kserve-7dbf875d88-dqxn5" podUID="a58c911f-e2eb-4820-a6b6-24afe78b5fd4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:14:33.064488 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:33.028610    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb0b93/simulator-w6-fb0b93-kserve-55b784b479-588bn" podUID="951a6f7e-a22e-40f2-ace2-1ce5d3bc9b49" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:14:33.064488 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:33.060691    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-65e59f/premium-w4-65e59f-kserve-6dc49468f8-rlzhx" podUID="8590a783-eb24-4759-ad74-1369f36aa9d2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:14:33.064488 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:33.061047    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-65e59f/distinct-2-w4-65e59f-kserve-689566757d-bf98q" podUID="64e66fdb-056e-4086-a85a-8c02613584a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": context deadline exceeded"
Oct 30 22:14:33.064488 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:33.062115    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7bb1c2/simulator-w3-7bb1c2-kserve-675645869c-vcwqv" podUID="1fce9daf-68c9-41d0-825b-2012bb93c8ba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:14:33.068323 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:33.068269    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779f32/premium-w0-779f32-kserve-67978797c8-jpkvl" podUID="40138fb9-9a75-4458-a351-11eb61c4a1e3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:14:33.069416 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:33.069377    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7bb1c2/distinct-w3-7bb1c2-kserve-8d695596c-8zgtw" podUID="36f0a6a2-54be-49bf-9202-502960cbb134" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:14:33.088456 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:33.062540    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-65e59f/simulator-w4-65e59f-kserve-85698bcd47-przch" podUID="ee5e843b-17be-4623-8c95-dc07c8b7f506" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Oct 30 22:14:33.243534 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:33.202700    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3acbb058-68zvq"]
Oct 30 22:14:33.492485 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:33.492252    5678 generic.go:356] "Generic (PLEG): container finished" podID="bde7f877-4938-42d4-be6c-cba04795ce83" containerID="026096dcf8dc7c8e0a89f9b9e951f419a9891095ba4a068e9cffe23d108aa87e" exitCode=0
Oct 30 22:14:33.492485 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:33.492415    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-9vxbk" event={"ID":"bde7f877-4938-42d4-be6c-cba04795ce83","Type":"ContainerDied","Data":"026096dcf8dc7c8e0a89f9b9e951f419a9891095ba4a068e9cffe23d108aa87e"}
Oct 30 22:14:33.494612 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:33.494584    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3acbb058-68zvq" event={"ID":"acfe6f2a-8f4e-4a7f-bde4-9486ed55cf07","Type":"ContainerStarted","Data":"4ed48a394680d66a39f6cd206d908f9137cf9f4c118195a6f31444b2919e5a05"}
Oct 30 22:14:33.587636 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:33.587536    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb0b93/distinct-2-w6-fb0b93-kserve-7d65bb4698-m6tkh" podUID="6fd276ef-522b-4bb4-b101-0a9c3fe9ecd3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:14:33.641543 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:33.641495    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7bb1c2/premium-w3-7bb1c2-kserve-7dbf875d88-dqxn5" podUID="a58c911f-e2eb-4820-a6b6-24afe78b5fd4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:14:33.644783 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:33.642655    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a00d9b/distinct-2-w1-a00d9b-kserve-68fd95957c-dmgnb" podUID="23a81fda-f7e3-4f91-8d96-9f19c7de6e09" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:14:33.655563 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:33.655536    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-7bb1c2/premium-w3-7bb1c2-kserve-7dbf875d88-dqxn5"
Oct 30 22:14:34.120426 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:34.119075    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7bb1c2/distinct-w3-7bb1c2-kserve-8d695596c-8zgtw" podUID="36f0a6a2-54be-49bf-9202-502960cbb134" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded"
Oct 30 22:14:34.120426 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:34.119547    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7bb1c2/simulator-w3-7bb1c2-kserve-675645869c-vcwqv" podUID="1fce9daf-68c9-41d0-825b-2012bb93c8ba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:14:34.120426 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:34.119804    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-7bb1c2/distinct-w3-7bb1c2-kserve-8d695596c-8zgtw"
Oct 30 22:14:34.120426 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:34.119863    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-7bb1c2/simulator-w3-7bb1c2-kserve-675645869c-vcwqv"
Oct 30 22:14:34.194434 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:34.184623    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9ndd9" podUID="81bd6415-f42f-446b-a64e-956f794b3956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": context deadline exceeded"
Oct 30 22:14:34.194434 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:34.184912    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-65e59f/premium-w4-65e59f-kserve-6dc49468f8-rlzhx" podUID="8590a783-eb24-4759-ad74-1369f36aa9d2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:14:34.194434 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:34.185089    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-65e59f/distinct-2-w4-65e59f-kserve-689566757d-bf98q" podUID="64e66fdb-056e-4086-a85a-8c02613584a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:14:34.194434 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:34.185242    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-65e59f/simulator-w4-65e59f-kserve-85698bcd47-przch" podUID="ee5e843b-17be-4623-8c95-dc07c8b7f506" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Oct 30 22:14:34.194434 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:34.185466    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-65e59f/premium-w4-65e59f-kserve-6dc49468f8-rlzhx"
Oct 30 22:14:34.194434 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:34.185524    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-65e59f/distinct-2-w4-65e59f-kserve-689566757d-bf98q"
Oct 30 22:14:34.194434 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:34.185561    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-65e59f/simulator-w4-65e59f-kserve-85698bcd47-przch"
Oct 30 22:14:34.237359 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:34.232068    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779f32/premium-w0-779f32-kserve-67978797c8-jpkvl" podUID="40138fb9-9a75-4458-a351-11eb61c4a1e3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Oct 30 22:14:34.237359 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:34.232369    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-779f32/premium-w0-779f32-kserve-67978797c8-jpkvl"
Oct 30 22:14:34.237359 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:34.232759    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb0b93/simulator-w6-fb0b93-kserve-55b784b479-588bn" podUID="951a6f7e-a22e-40f2-ace2-1ce5d3bc9b49" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 30 22:14:34.237359 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:34.232966    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-fb0b93/simulator-w6-fb0b93-kserve-55b784b479-588bn"
Oct 30 22:14:34.605493 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:34.598344    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb0b93/distinct-2-w6-fb0b93-kserve-7d65bb4698-m6tkh" podUID="6fd276ef-522b-4bb4-b101-0a9c3fe9ecd3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:14:34.657426 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:34.656724    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7bb1c2/premium-w3-7bb1c2-kserve-7dbf875d88-dqxn5" podUID="a58c911f-e2eb-4820-a6b6-24afe78b5fd4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:14:34.657426 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:34.656989    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a00d9b/distinct-2-w1-a00d9b-kserve-68fd95957c-dmgnb" podUID="23a81fda-f7e3-4f91-8d96-9f19c7de6e09" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:14:34.936425 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:34.910211    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-fb0b93/distinct-2-w6-fb0b93-kserve-7d65bb4698-m6tkh"
Oct 30 22:14:34.985810 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:34.985780    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a00d9b/distinct-2-w1-a00d9b-kserve-68fd95957c-dmgnb"
Oct 30 22:14:35.148484 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:35.123081    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7bb1c2/simulator-w3-7bb1c2-kserve-675645869c-vcwqv" podUID="1fce9daf-68c9-41d0-825b-2012bb93c8ba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:14:35.280910 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:35.132765    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7bb1c2/distinct-w3-7bb1c2-kserve-8d695596c-8zgtw" podUID="36f0a6a2-54be-49bf-9202-502960cbb134" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:14:35.281319 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:35.204811    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-65e59f/simulator-w4-65e59f-kserve-85698bcd47-przch" podUID="ee5e843b-17be-4623-8c95-dc07c8b7f506" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:14:35.281705 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:35.204847    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-65e59f/premium-w4-65e59f-kserve-6dc49468f8-rlzhx" podUID="8590a783-eb24-4759-ad74-1369f36aa9d2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:14:35.306599 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:35.204868    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-65e59f/distinct-2-w4-65e59f-kserve-689566757d-bf98q" podUID="64e66fdb-056e-4086-a85a-8c02613584a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:14:35.362697 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:35.204912    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9ndd9" podUID="81bd6415-f42f-446b-a64e-956f794b3956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:14:35.371558 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:35.257852    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb0b93/simulator-w6-fb0b93-kserve-55b784b479-588bn" podUID="951a6f7e-a22e-40f2-ace2-1ce5d3bc9b49" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:14:35.388425 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:35.376771    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9ndd9"
Oct 30 22:14:35.414482 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:35.257894    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779f32/premium-w0-779f32-kserve-67978797c8-jpkvl" podUID="40138fb9-9a75-4458-a351-11eb61c4a1e3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Oct 30 22:14:35.585832 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:35.585793    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-9vxbk"
Oct 30 22:14:35.699487 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:35.675434    5678 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-65e59f/simulator-w4-65e59f-kserve-85698bcd47-przch" podUID="ee5e843b-17be-4623-8c95-dc07c8b7f506" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": context deadline exceeded"
Oct 30 22:14:35.699487 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:35.676920    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb0b93/distinct-2-w6-fb0b93-kserve-7d65bb4698-m6tkh" podUID="6fd276ef-522b-4bb4-b101-0a9c3fe9ecd3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": context deadline exceeded"
Oct 30 22:14:35.699487 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:35.678756    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a00d9b/distinct-2-w1-a00d9b-kserve-68fd95957c-dmgnb" podUID="23a81fda-f7e3-4f91-8d96-9f19c7de6e09" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:14:35.730439 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:35.700443    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7bb1c2/premium-w3-7bb1c2-kserve-7dbf875d88-dqxn5" podUID="a58c911f-e2eb-4820-a6b6-24afe78b5fd4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Oct 30 22:14:35.837432 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:35.800991    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bde7f877-4938-42d4-be6c-cba04795ce83-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bde7f877-4938-42d4-be6c-cba04795ce83-workload-certs\") pod \"bde7f877-4938-42d4-be6c-cba04795ce83\" (UID: \"bde7f877-4938-42d4-be6c-cba04795ce83\") "
Oct 30 22:14:35.854637 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:35.854601    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bde7f877-4938-42d4-be6c-cba04795ce83-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bde7f877-4938-42d4-be6c-cba04795ce83-istio-data\") pod \"bde7f877-4938-42d4-be6c-cba04795ce83\" (UID: \"bde7f877-4938-42d4-be6c-cba04795ce83\") "
Oct 30 22:14:35.858626 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:35.858598    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bde7f877-4938-42d4-be6c-cba04795ce83-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bde7f877-4938-42d4-be6c-cba04795ce83-workload-socket\") pod \"bde7f877-4938-42d4-be6c-cba04795ce83\" (UID: \"bde7f877-4938-42d4-be6c-cba04795ce83\") "
Oct 30 22:14:35.886073 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:35.886039    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bde7f877-4938-42d4-be6c-cba04795ce83-istio-token\" (UniqueName: \"kubernetes.io/projected/bde7f877-4938-42d4-be6c-cba04795ce83-istio-token\") pod \"bde7f877-4938-42d4-be6c-cba04795ce83\" (UID: \"bde7f877-4938-42d4-be6c-cba04795ce83\") "
Oct 30 22:14:35.886740 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:35.886714    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bde7f877-4938-42d4-be6c-cba04795ce83-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bde7f877-4938-42d4-be6c-cba04795ce83-istiod-ca-cert\") pod \"bde7f877-4938-42d4-be6c-cba04795ce83\" (UID: \"bde7f877-4938-42d4-be6c-cba04795ce83\") "
Oct 30 22:14:35.935567 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:35.935525    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bde7f877-4938-42d4-be6c-cba04795ce83-kube-api-access-mx68z\" (UniqueName: \"kubernetes.io/projected/bde7f877-4938-42d4-be6c-cba04795ce83-kube-api-access-mx68z\") pod \"bde7f877-4938-42d4-be6c-cba04795ce83\" (UID: \"bde7f877-4938-42d4-be6c-cba04795ce83\") "
Oct 30 22:14:35.935839 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:35.935814    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bde7f877-4938-42d4-be6c-cba04795ce83-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bde7f877-4938-42d4-be6c-cba04795ce83-istio-podinfo\") pod \"bde7f877-4938-42d4-be6c-cba04795ce83\" (UID: \"bde7f877-4938-42d4-be6c-cba04795ce83\") "
Oct 30 22:14:35.935998 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:35.935982    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bde7f877-4938-42d4-be6c-cba04795ce83-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bde7f877-4938-42d4-be6c-cba04795ce83-istio-envoy\") pod \"bde7f877-4938-42d4-be6c-cba04795ce83\" (UID: \"bde7f877-4938-42d4-be6c-cba04795ce83\") "
Oct 30 22:14:35.936149 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:35.936131    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bde7f877-4938-42d4-be6c-cba04795ce83-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bde7f877-4938-42d4-be6c-cba04795ce83-credential-socket\") pod \"bde7f877-4938-42d4-be6c-cba04795ce83\" (UID: \"bde7f877-4938-42d4-be6c-cba04795ce83\") "
Oct 30 22:14:35.940457 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:35.850494    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3acbb058-68zvq" event={"ID":"acfe6f2a-8f4e-4a7f-bde4-9486ed55cf07","Type":"ContainerDied","Data":"d3302a639532d27ee202582e86eef078337558fab6447c1bc794c0e7ab66bce0"}
Oct 30 22:14:35.941225 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:35.803269    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bde7f877-4938-42d4-be6c-cba04795ce83-workload-certs" pod "bde7f877-4938-42d4-be6c-cba04795ce83" (UID: "bde7f877-4938-42d4-be6c-cba04795ce83"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:35.941354 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:35.858492    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bde7f877-4938-42d4-be6c-cba04795ce83-istio-data" pod "bde7f877-4938-42d4-be6c-cba04795ce83" (UID: "bde7f877-4938-42d4-be6c-cba04795ce83"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:35.941456 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:35.908633    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bde7f877-4938-42d4-be6c-cba04795ce83-istiod-ca-cert" pod "bde7f877-4938-42d4-be6c-cba04795ce83" (UID: "bde7f877-4938-42d4-be6c-cba04795ce83"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:14:35.941560 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:35.909007    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bde7f877-4938-42d4-be6c-cba04795ce83-workload-socket" pod "bde7f877-4938-42d4-be6c-cba04795ce83" (UID: "bde7f877-4938-42d4-be6c-cba04795ce83"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:35.941640 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:35.850463    5678 generic.go:356] "Generic (PLEG): container finished" podID="acfe6f2a-8f4e-4a7f-bde4-9486ed55cf07" containerID="d3302a639532d27ee202582e86eef078337558fab6447c1bc794c0e7ab66bce0" exitCode=0
Oct 30 22:14:35.969914 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:35.962323    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bde7f877-4938-42d4-be6c-cba04795ce83-credential-socket" pod "bde7f877-4938-42d4-be6c-cba04795ce83" (UID: "bde7f877-4938-42d4-be6c-cba04795ce83"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:36.173426 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:36.146582    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bde7f877-4938-42d4-be6c-cba04795ce83-istio-podinfo" pod "bde7f877-4938-42d4-be6c-cba04795ce83" (UID: "bde7f877-4938-42d4-be6c-cba04795ce83"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:14:36.200778 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:36.200745    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bde7f877-4938-42d4-be6c-cba04795ce83-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bde7f877-4938-42d4-be6c-cba04795ce83-istio-podinfo\") pod \"bde7f877-4938-42d4-be6c-cba04795ce83\" (UID: \"bde7f877-4938-42d4-be6c-cba04795ce83\") "
Oct 30 22:14:36.204456 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:14:36.201116    5678 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/bde7f877-4938-42d4-be6c-cba04795ce83/volumes/kubernetes.io~downward-api/istio-podinfo
Oct 30 22:14:36.225654 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:36.208480    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bde7f877-4938-42d4-be6c-cba04795ce83-istio-podinfo" pod "bde7f877-4938-42d4-be6c-cba04795ce83" (UID: "bde7f877-4938-42d4-be6c-cba04795ce83"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:14:36.226183 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:36.221523    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bde7f877-4938-42d4-be6c-cba04795ce83-istio-envoy" pod "bde7f877-4938-42d4-be6c-cba04795ce83" (UID: "bde7f877-4938-42d4-be6c-cba04795ce83"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:36.226343 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:36.223543    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bde7f877-4938-42d4-be6c-cba04795ce83-istio-token" pod "bde7f877-4938-42d4-be6c-cba04795ce83" (UID: "bde7f877-4938-42d4-be6c-cba04795ce83"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:14:36.226968 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:36.226944    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bde7f877-4938-42d4-be6c-cba04795ce83-istio-token\" (UniqueName: \"kubernetes.io/projected/bde7f877-4938-42d4-be6c-cba04795ce83-istio-token\") pod \"bde7f877-4938-42d4-be6c-cba04795ce83\" (UID: \"bde7f877-4938-42d4-be6c-cba04795ce83\") "
Oct 30 22:14:36.227989 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:36.227968    5678 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bde7f877-4938-42d4-be6c-cba04795ce83-workload-certs\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:14:36.228147 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:36.228133    5678 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bde7f877-4938-42d4-be6c-cba04795ce83-istio-data\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:14:36.228275 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:36.228261    5678 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bde7f877-4938-42d4-be6c-cba04795ce83-workload-socket\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:14:36.228386 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:36.228372    5678 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bde7f877-4938-42d4-be6c-cba04795ce83-istiod-ca-cert\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:14:36.228497 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:36.228483    5678 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bde7f877-4938-42d4-be6c-cba04795ce83-istio-podinfo\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:14:36.228623 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:36.228609    5678 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bde7f877-4938-42d4-be6c-cba04795ce83-istio-envoy\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:14:36.228745 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:36.228732    5678 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bde7f877-4938-42d4-be6c-cba04795ce83-credential-socket\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:14:36.228936 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:14:36.228923    5678 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/bde7f877-4938-42d4-be6c-cba04795ce83/volumes/kubernetes.io~projected/istio-token
Oct 30 22:14:36.229108 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:36.229080    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bde7f877-4938-42d4-be6c-cba04795ce83-istio-token" pod "bde7f877-4938-42d4-be6c-cba04795ce83" (UID: "bde7f877-4938-42d4-be6c-cba04795ce83"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:14:36.331015 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:36.304536    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-65e59f/simulator-w4-65e59f-kserve-85698bcd47-przch" podUID="ee5e843b-17be-4623-8c95-dc07c8b7f506" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:14:36.374428 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:36.331557    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7bb1c2/distinct-w3-7bb1c2-kserve-8d695596c-8zgtw" podUID="36f0a6a2-54be-49bf-9202-502960cbb134" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:14:36.374428 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:36.331900    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7bb1c2/simulator-w3-7bb1c2-kserve-675645869c-vcwqv" podUID="1fce9daf-68c9-41d0-825b-2012bb93c8ba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Oct 30 22:14:36.374428 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:36.333383    5678 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bde7f877-4938-42d4-be6c-cba04795ce83-istio-token\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:14:36.374428 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:36.333454    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-65e59f/premium-w4-65e59f-kserve-6dc49468f8-rlzhx" podUID="8590a783-eb24-4759-ad74-1369f36aa9d2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:14:36.374428 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:36.373455    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9ndd9" podUID="81bd6415-f42f-446b-a64e-956f794b3956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:14:36.375630 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:36.375597    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-65e59f/distinct-2-w4-65e59f-kserve-689566757d-bf98q" podUID="64e66fdb-056e-4086-a85a-8c02613584a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": context deadline exceeded"
Oct 30 22:14:36.400497 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:36.400455    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb0b93/simulator-w6-fb0b93-kserve-55b784b479-588bn" podUID="951a6f7e-a22e-40f2-ace2-1ce5d3bc9b49" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:14:36.441383 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:36.430751    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779f32/premium-w0-779f32-kserve-67978797c8-jpkvl" podUID="40138fb9-9a75-4458-a351-11eb61c4a1e3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:14:36.622524 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:36.567167    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bde7f877-4938-42d4-be6c-cba04795ce83-kube-api-access-mx68z" pod "bde7f877-4938-42d4-be6c-cba04795ce83" (UID: "bde7f877-4938-42d4-be6c-cba04795ce83"). InnerVolumeSpecName "kube-api-access-mx68z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:14:36.702426 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:36.698695    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a00d9b/distinct-2-w1-a00d9b-kserve-68fd95957c-dmgnb" podUID="23a81fda-f7e3-4f91-8d96-9f19c7de6e09" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:14:36.702426 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:36.698987    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb0b93/distinct-2-w6-fb0b93-kserve-7d65bb4698-m6tkh" podUID="6fd276ef-522b-4bb4-b101-0a9c3fe9ecd3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": context deadline exceeded"
Oct 30 22:14:36.742426 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:36.704779    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7bb1c2/premium-w3-7bb1c2-kserve-7dbf875d88-dqxn5" podUID="a58c911f-e2eb-4820-a6b6-24afe78b5fd4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:14:36.742426 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:36.742090    5678 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9ndd9" podUID="81bd6415-f42f-446b-a64e-956f794b3956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:14:36.800431 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:36.779495    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mx68z\" (UniqueName: \"kubernetes.io/projected/bde7f877-4938-42d4-be6c-cba04795ce83-kube-api-access-mx68z\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:14:37.252040 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:37.251991    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-7bb1c2/distinct-w3-7bb1c2-kserve-8d695596c-8zgtw"
Oct 30 22:14:37.261790 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:37.261765    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-779f32/premium-w0-779f32-kserve-67978797c8-jpkvl"
Oct 30 22:14:37.262027 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:37.262006    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-65e59f/distinct-2-w4-65e59f-kserve-689566757d-bf98q"
Oct 30 22:14:37.262147 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:37.262054    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9ndd9"
Oct 30 22:14:37.262147 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:37.262087    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-65e59f/premium-w4-65e59f-kserve-6dc49468f8-rlzhx"
Oct 30 22:14:37.262147 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:37.262120    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-65e59f/simulator-w4-65e59f-kserve-85698bcd47-przch"
Oct 30 22:14:37.262747 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:37.262711    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a00d9b/distinct-2-w1-a00d9b-kserve-68fd95957c-dmgnb"
Oct 30 22:14:37.262874 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:37.262761    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-7bb1c2/premium-w3-7bb1c2-kserve-7dbf875d88-dqxn5"
Oct 30 22:14:37.262874 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:37.262793    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-fb0b93/simulator-w6-fb0b93-kserve-55b784b479-588bn"
Oct 30 22:14:37.263634 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:37.263616    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-fb0b93/distinct-2-w6-fb0b93-kserve-7d65bb4698-m6tkh"
Oct 30 22:14:37.264660 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:37.264637    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-7bb1c2/simulator-w3-7bb1c2-kserve-675645869c-vcwqv"
Oct 30 22:14:37.265081 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:37.265055    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-9vxbk"
Oct 30 22:14:37.265217 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:37.265084    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-9vxbk" event={"ID":"bde7f877-4938-42d4-be6c-cba04795ce83","Type":"ContainerDied","Data":"99f73a7de5fb3c5c88d28ff7d650bceeb57930e2af6a19d8cd16f573f6f46c7c"}
Oct 30 22:14:37.265310 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:37.265251    5678 scope.go:122] "RemoveContainer" containerID="026096dcf8dc7c8e0a89f9b9e951f419a9891095ba4a068e9cffe23d108aa87e"
Oct 30 22:14:37.446353 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:37.446274    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3acbb058-68zvq"
Oct 30 22:14:37.467172 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:37.467144    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acfe6f2a-8f4e-4a7f-bde4-9486ed55cf07-kube-api-access-g84x2\" (UniqueName: \"kubernetes.io/projected/acfe6f2a-8f4e-4a7f-bde4-9486ed55cf07-kube-api-access-g84x2\") pod \"acfe6f2a-8f4e-4a7f-bde4-9486ed55cf07\" (UID: \"acfe6f2a-8f4e-4a7f-bde4-9486ed55cf07\") "
Oct 30 22:14:37.467387 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:37.467180    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/acfe6f2a-8f4e-4a7f-bde4-9486ed55cf07-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acfe6f2a-8f4e-4a7f-bde4-9486ed55cf07-maas-api-openshift-ca\") pod \"acfe6f2a-8f4e-4a7f-bde4-9486ed55cf07\" (UID: \"acfe6f2a-8f4e-4a7f-bde4-9486ed55cf07\") "
Oct 30 22:14:37.467387 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:37.467361    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/acfe6f2a-8f4e-4a7f-bde4-9486ed55cf07-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acfe6f2a-8f4e-4a7f-bde4-9486ed55cf07-maas-api-xks-ca\") pod \"acfe6f2a-8f4e-4a7f-bde4-9486ed55cf07\" (UID: \"acfe6f2a-8f4e-4a7f-bde4-9486ed55cf07\") "
Oct 30 22:14:37.467584 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:37.467563    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acfe6f2a-8f4e-4a7f-bde4-9486ed55cf07-maas-api-openshift-ca" pod "acfe6f2a-8f4e-4a7f-bde4-9486ed55cf07" (UID: "acfe6f2a-8f4e-4a7f-bde4-9486ed55cf07"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:14:37.467986 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:37.467967    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acfe6f2a-8f4e-4a7f-bde4-9486ed55cf07-maas-api-openshift-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:14:37.469170 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:37.469151    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/acfe6f2a-8f4e-4a7f-bde4-9486ed55cf07-maas-api-xks-ca" pod "acfe6f2a-8f4e-4a7f-bde4-9486ed55cf07" (UID: "acfe6f2a-8f4e-4a7f-bde4-9486ed55cf07"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:14:37.469282 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:37.469264    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acfe6f2a-8f4e-4a7f-bde4-9486ed55cf07-kube-api-access-g84x2" pod "acfe6f2a-8f4e-4a7f-bde4-9486ed55cf07" (UID: "acfe6f2a-8f4e-4a7f-bde4-9486ed55cf07"). InnerVolumeSpecName "kube-api-access-g84x2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:14:37.569473 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:37.569443    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acfe6f2a-8f4e-4a7f-bde4-9486ed55cf07-maas-api-xks-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:14:37.569473 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:37.569468    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g84x2\" (UniqueName: \"kubernetes.io/projected/acfe6f2a-8f4e-4a7f-bde4-9486ed55cf07-kube-api-access-g84x2\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:14:38.276907 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:38.276877    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3acbb058-68zvq"
Oct 30 22:14:38.276907 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:38.276898    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3acbb058-68zvq" event={"ID":"acfe6f2a-8f4e-4a7f-bde4-9486ed55cf07","Type":"ContainerDied","Data":"4ed48a394680d66a39f6cd206d908f9137cf9f4c118195a6f31444b2919e5a05"}
Oct 30 22:14:38.277493 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:38.276954    5678 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4ed48a394680d66a39f6cd206d908f9137cf9f4c118195a6f31444b2919e5a05"
Oct 30 22:14:38.567005 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:38.566911    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-9vxbk"]
Oct 30 22:14:38.613521 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:38.613485    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-9vxbk"]
Oct 30 22:14:40.106000 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:40.105972    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0794c9d6-jjsx9"]
Oct 30 22:14:40.151978 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:40.151946    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0794c9d6-jjsx9"]
Oct 30 22:14:40.525322 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:40.525288    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="add39a5f-865c-4416-9877-d2fcbe815b47" path="/var/lib/kubelet/pods/add39a5f-865c-4416-9877-d2fcbe815b47/volumes"
Oct 30 22:14:40.525833 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:40.525817    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bde7f877-4938-42d4-be6c-cba04795ce83" path="/var/lib/kubelet/pods/bde7f877-4938-42d4-be6c-cba04795ce83/volumes"
Oct 30 22:14:40.573047 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:14:40.573011    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 22:14:40.836736 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:40.836606    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83996232-dxszx"]
Oct 30 22:14:40.844929 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:40.837742    5678 cpu_manager.go:414] "containerMap: removing container" podUID="bde7f877-4938-42d4-be6c-cba04795ce83" containerName="istio-proxy"
Oct 30 22:14:40.844929 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:40.837766    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bde7f877-4938-42d4-be6c-cba04795ce83" containerName="istio-proxy"
Oct 30 22:14:40.844929 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:40.837803    5678 cpu_manager.go:414] "containerMap: removing container" podUID="acfe6f2a-8f4e-4a7f-bde4-9486ed55cf07" containerName="revoke-keys"
Oct 30 22:14:40.844929 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:40.837811    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acfe6f2a-8f4e-4a7f-bde4-9486ed55cf07" containerName="revoke-keys"
Oct 30 22:14:40.844929 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:40.837992    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="acfe6f2a-8f4e-4a7f-bde4-9486ed55cf07" containerName="revoke-keys"
Oct 30 22:14:40.844929 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:40.838024    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="bde7f877-4938-42d4-be6c-cba04795ce83" containerName="istio-proxy"
Oct 30 22:14:40.849047 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:40.849022    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83996232-dxszx"
Oct 30 22:14:40.897638 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:40.897604    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83996232-dxszx"]
Oct 30 22:14:40.929134 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:40.929097    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/506bf715-6464-4d0f-a5a7-85ad36039f21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83996232-dxszx\" (UID: \"506bf715-6464-4d0f-a5a7-85ad36039f21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83996232-dxszx"
Oct 30 22:14:40.929300 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:40.929150    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/506bf715-6464-4d0f-a5a7-85ad36039f21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83996232-dxszx\" (UID: \"506bf715-6464-4d0f-a5a7-85ad36039f21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83996232-dxszx"
Oct 30 22:14:40.929351 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:40.929307    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cppd\" (UniqueName: \"kubernetes.io/projected/506bf715-6464-4d0f-a5a7-85ad36039f21-kube-api-access-9cppd\") pod \"maas-api-revoke-sub-83996232-dxszx\" (UID: \"506bf715-6464-4d0f-a5a7-85ad36039f21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83996232-dxszx"
Oct 30 22:14:40.937882 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:40.937844    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:14:40.938048 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:40.937943    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5rvcj\""
Oct 30 22:14:41.030297 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:41.030258    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/506bf715-6464-4d0f-a5a7-85ad36039f21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83996232-dxszx\" (UID: \"506bf715-6464-4d0f-a5a7-85ad36039f21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83996232-dxszx"
Oct 30 22:14:41.030520 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:41.030444    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cppd\" (UniqueName: \"kubernetes.io/projected/506bf715-6464-4d0f-a5a7-85ad36039f21-kube-api-access-9cppd\") pod \"maas-api-revoke-sub-83996232-dxszx\" (UID: \"506bf715-6464-4d0f-a5a7-85ad36039f21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83996232-dxszx"
Oct 30 22:14:41.030693 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:41.030668    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/506bf715-6464-4d0f-a5a7-85ad36039f21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83996232-dxszx\" (UID: \"506bf715-6464-4d0f-a5a7-85ad36039f21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83996232-dxszx"
Oct 30 22:14:41.031321 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:41.031299    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/506bf715-6464-4d0f-a5a7-85ad36039f21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83996232-dxszx\" (UID: \"506bf715-6464-4d0f-a5a7-85ad36039f21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83996232-dxszx"
Oct 30 22:14:41.032670 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:41.032648    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/506bf715-6464-4d0f-a5a7-85ad36039f21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83996232-dxszx\" (UID: \"506bf715-6464-4d0f-a5a7-85ad36039f21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83996232-dxszx"
Oct 30 22:14:41.089459 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:41.089363    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cppd\" (UniqueName: \"kubernetes.io/projected/506bf715-6464-4d0f-a5a7-85ad36039f21-kube-api-access-9cppd\") pod \"maas-api-revoke-sub-83996232-dxszx\" (UID: \"506bf715-6464-4d0f-a5a7-85ad36039f21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83996232-dxszx"
Oct 30 22:14:41.164175 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:41.164142    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83996232-dxszx"
Oct 30 22:14:41.565944 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:14:41.553006    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod506bf715_6464_4d0f_a5a7_85ad36039f21.slice/crio-262c768e0f111b32ce90c45196c3ddd5dd6b94f41d62cb90e60de137419c2b63 WatchSource:0}: Error finding container 262c768e0f111b32ce90c45196c3ddd5dd6b94f41d62cb90e60de137419c2b63: Status 404 returned error can't find the container with id 262c768e0f111b32ce90c45196c3ddd5dd6b94f41d62cb90e60de137419c2b63
Oct 30 22:14:41.565944 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:41.554136    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83996232-dxszx"]
Oct 30 22:14:42.324585 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:42.324550    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83996232-dxszx" event={"ID":"506bf715-6464-4d0f-a5a7-85ad36039f21","Type":"ContainerStarted","Data":"b2baf0fdf7d7d08e4267d9745aaa28b8623dc6d647d4dfa89f24cc0b0af05fbc"}
Oct 30 22:14:42.324969 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:42.324609    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83996232-dxszx" event={"ID":"506bf715-6464-4d0f-a5a7-85ad36039f21","Type":"ContainerStarted","Data":"262c768e0f111b32ce90c45196c3ddd5dd6b94f41d62cb90e60de137419c2b63"}
Oct 30 22:14:42.437906 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:42.437854    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83996232-dxszx" podStartSLOduration=2.437840926 podStartE2EDuration="2.437840926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:14:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:14:42.436358648 +0000 UTC m=+1460.040666212" watchObservedRunningTime="2026-10-30 22:14:42.437840926 +0000 UTC m=+1460.042148479"
Oct 30 22:14:43.334513 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:43.334479    5678 generic.go:356] "Generic (PLEG): container finished" podID="506bf715-6464-4d0f-a5a7-85ad36039f21" containerID="b2baf0fdf7d7d08e4267d9745aaa28b8623dc6d647d4dfa89f24cc0b0af05fbc" exitCode=0
Oct 30 22:14:43.334880 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:43.334522    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83996232-dxszx" event={"ID":"506bf715-6464-4d0f-a5a7-85ad36039f21","Type":"ContainerDied","Data":"b2baf0fdf7d7d08e4267d9745aaa28b8623dc6d647d4dfa89f24cc0b0af05fbc"}
Oct 30 22:14:44.527929 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:44.527904    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83996232-dxszx"
Oct 30 22:14:44.585714 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:44.585688    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/506bf715-6464-4d0f-a5a7-85ad36039f21-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/506bf715-6464-4d0f-a5a7-85ad36039f21-maas-api-openshift-ca\") pod \"506bf715-6464-4d0f-a5a7-85ad36039f21\" (UID: \"506bf715-6464-4d0f-a5a7-85ad36039f21\") "
Oct 30 22:14:44.585854 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:44.585726    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/506bf715-6464-4d0f-a5a7-85ad36039f21-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/506bf715-6464-4d0f-a5a7-85ad36039f21-maas-api-xks-ca\") pod \"506bf715-6464-4d0f-a5a7-85ad36039f21\" (UID: \"506bf715-6464-4d0f-a5a7-85ad36039f21\") "
Oct 30 22:14:44.585854 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:44.585799    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/506bf715-6464-4d0f-a5a7-85ad36039f21-kube-api-access-9cppd\" (UniqueName: \"kubernetes.io/projected/506bf715-6464-4d0f-a5a7-85ad36039f21-kube-api-access-9cppd\") pod \"506bf715-6464-4d0f-a5a7-85ad36039f21\" (UID: \"506bf715-6464-4d0f-a5a7-85ad36039f21\") "
Oct 30 22:14:44.586055 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:44.586029    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/506bf715-6464-4d0f-a5a7-85ad36039f21-maas-api-openshift-ca" pod "506bf715-6464-4d0f-a5a7-85ad36039f21" (UID: "506bf715-6464-4d0f-a5a7-85ad36039f21"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:14:44.586692 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:44.586667    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/506bf715-6464-4d0f-a5a7-85ad36039f21-maas-api-openshift-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:14:44.587873 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:44.587848    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/506bf715-6464-4d0f-a5a7-85ad36039f21-maas-api-xks-ca" pod "506bf715-6464-4d0f-a5a7-85ad36039f21" (UID: "506bf715-6464-4d0f-a5a7-85ad36039f21"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:14:44.587976 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:44.587904    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/506bf715-6464-4d0f-a5a7-85ad36039f21-kube-api-access-9cppd" pod "506bf715-6464-4d0f-a5a7-85ad36039f21" (UID: "506bf715-6464-4d0f-a5a7-85ad36039f21"). InnerVolumeSpecName "kube-api-access-9cppd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:14:44.687716 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:44.687682    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/506bf715-6464-4d0f-a5a7-85ad36039f21-maas-api-xks-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:14:44.687716 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:44.687711    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cppd\" (UniqueName: \"kubernetes.io/projected/506bf715-6464-4d0f-a5a7-85ad36039f21-kube-api-access-9cppd\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:14:45.351808 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:45.351777    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83996232-dxszx"
Oct 30 22:14:45.351808 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:45.351798    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83996232-dxszx" event={"ID":"506bf715-6464-4d0f-a5a7-85ad36039f21","Type":"ContainerDied","Data":"262c768e0f111b32ce90c45196c3ddd5dd6b94f41d62cb90e60de137419c2b63"}
Oct 30 22:14:45.352036 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:45.351847    5678 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="262c768e0f111b32ce90c45196c3ddd5dd6b94f41d62cb90e60de137419c2b63"
Oct 30 22:14:51.589700 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:51.589667    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53f4fd0e-rwrt9"]
Oct 30 22:14:51.590331 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:51.590317    5678 cpu_manager.go:414] "containerMap: removing container" podUID="506bf715-6464-4d0f-a5a7-85ad36039f21" containerName="revoke-keys"
Oct 30 22:14:51.590383 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:51.590332    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="506bf715-6464-4d0f-a5a7-85ad36039f21" containerName="revoke-keys"
Oct 30 22:14:51.590463 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:51.590453    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="506bf715-6464-4d0f-a5a7-85ad36039f21" containerName="revoke-keys"
Oct 30 22:14:51.595067 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:51.595050    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53f4fd0e-rwrt9"
Oct 30 22:14:51.647996 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:51.647962    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53f4fd0e-rwrt9"]
Oct 30 22:14:51.678573 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:51.678543    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0818834a-f7a8-4a39-97c4-159253f365df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-53f4fd0e-rwrt9\" (UID: \"0818834a-f7a8-4a39-97c4-159253f365df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53f4fd0e-rwrt9"
Oct 30 22:14:51.678738 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:51.678612    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0818834a-f7a8-4a39-97c4-159253f365df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-53f4fd0e-rwrt9\" (UID: \"0818834a-f7a8-4a39-97c4-159253f365df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53f4fd0e-rwrt9"
Oct 30 22:14:51.678851 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:51.678824    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bmmh\" (UniqueName: \"kubernetes.io/projected/0818834a-f7a8-4a39-97c4-159253f365df-kube-api-access-6bmmh\") pod \"maas-api-revoke-sub-53f4fd0e-rwrt9\" (UID: \"0818834a-f7a8-4a39-97c4-159253f365df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53f4fd0e-rwrt9"
Oct 30 22:14:51.688879 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:51.688853    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:14:51.688991 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:51.688916    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5rvcj\""
Oct 30 22:14:51.780051 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:51.780011    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0818834a-f7a8-4a39-97c4-159253f365df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-53f4fd0e-rwrt9\" (UID: \"0818834a-f7a8-4a39-97c4-159253f365df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53f4fd0e-rwrt9"
Oct 30 22:14:51.780253 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:51.780078    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0818834a-f7a8-4a39-97c4-159253f365df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-53f4fd0e-rwrt9\" (UID: \"0818834a-f7a8-4a39-97c4-159253f365df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53f4fd0e-rwrt9"
Oct 30 22:14:51.780253 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:51.780143    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bmmh\" (UniqueName: \"kubernetes.io/projected/0818834a-f7a8-4a39-97c4-159253f365df-kube-api-access-6bmmh\") pod \"maas-api-revoke-sub-53f4fd0e-rwrt9\" (UID: \"0818834a-f7a8-4a39-97c4-159253f365df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53f4fd0e-rwrt9"
Oct 30 22:14:51.780763 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:51.780737    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0818834a-f7a8-4a39-97c4-159253f365df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-53f4fd0e-rwrt9\" (UID: \"0818834a-f7a8-4a39-97c4-159253f365df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53f4fd0e-rwrt9"
Oct 30 22:14:51.782381 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:51.782362    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0818834a-f7a8-4a39-97c4-159253f365df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-53f4fd0e-rwrt9\" (UID: \"0818834a-f7a8-4a39-97c4-159253f365df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53f4fd0e-rwrt9"
Oct 30 22:14:51.841651 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:51.841580    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bmmh\" (UniqueName: \"kubernetes.io/projected/0818834a-f7a8-4a39-97c4-159253f365df-kube-api-access-6bmmh\") pod \"maas-api-revoke-sub-53f4fd0e-rwrt9\" (UID: \"0818834a-f7a8-4a39-97c4-159253f365df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53f4fd0e-rwrt9"
Oct 30 22:14:51.908772 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:51.908739    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53f4fd0e-rwrt9"
Oct 30 22:14:52.298140 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:14:52.298111    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0818834a_f7a8_4a39_97c4_159253f365df.slice/crio-6eb68382fdb86c7c8dc9aaeb27ff28b703f8bae02ca719dbc3d70124f0d86a32 WatchSource:0}: Error finding container 6eb68382fdb86c7c8dc9aaeb27ff28b703f8bae02ca719dbc3d70124f0d86a32: Status 404 returned error can't find the container with id 6eb68382fdb86c7c8dc9aaeb27ff28b703f8bae02ca719dbc3d70124f0d86a32
Oct 30 22:14:52.299123 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:52.299098    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53f4fd0e-rwrt9"]
Oct 30 22:14:52.414055 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:52.414023    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53f4fd0e-rwrt9" event={"ID":"0818834a-f7a8-4a39-97c4-159253f365df","Type":"ContainerStarted","Data":"6eb68382fdb86c7c8dc9aaeb27ff28b703f8bae02ca719dbc3d70124f0d86a32"}
Oct 30 22:14:53.424468 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:53.424422    5678 generic.go:356] "Generic (PLEG): container finished" podID="0818834a-f7a8-4a39-97c4-159253f365df" containerID="07a863ec3e008a3ed94dd22b5b8578c5a1b1b85e0a1ec9ebb7291f67d8fca16b" exitCode=0
Oct 30 22:14:53.424842 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:53.424461    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53f4fd0e-rwrt9" event={"ID":"0818834a-f7a8-4a39-97c4-159253f365df","Type":"ContainerDied","Data":"07a863ec3e008a3ed94dd22b5b8578c5a1b1b85e0a1ec9ebb7291f67d8fca16b"}
Oct 30 22:14:54.600591 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:54.600564    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53f4fd0e-rwrt9"
Oct 30 22:14:54.728867 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:54.728784    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0818834a-f7a8-4a39-97c4-159253f365df-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0818834a-f7a8-4a39-97c4-159253f365df-maas-api-xks-ca\") pod \"0818834a-f7a8-4a39-97c4-159253f365df\" (UID: \"0818834a-f7a8-4a39-97c4-159253f365df\") "
Oct 30 22:14:54.729023 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:54.728874    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0818834a-f7a8-4a39-97c4-159253f365df-kube-api-access-6bmmh\" (UniqueName: \"kubernetes.io/projected/0818834a-f7a8-4a39-97c4-159253f365df-kube-api-access-6bmmh\") pod \"0818834a-f7a8-4a39-97c4-159253f365df\" (UID: \"0818834a-f7a8-4a39-97c4-159253f365df\") "
Oct 30 22:14:54.729023 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:54.729011    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0818834a-f7a8-4a39-97c4-159253f365df-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0818834a-f7a8-4a39-97c4-159253f365df-maas-api-openshift-ca\") pod \"0818834a-f7a8-4a39-97c4-159253f365df\" (UID: \"0818834a-f7a8-4a39-97c4-159253f365df\") "
Oct 30 22:14:54.729385 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:54.729355    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0818834a-f7a8-4a39-97c4-159253f365df-maas-api-openshift-ca" pod "0818834a-f7a8-4a39-97c4-159253f365df" (UID: "0818834a-f7a8-4a39-97c4-159253f365df"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:14:54.729835 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:54.729810    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0818834a-f7a8-4a39-97c4-159253f365df-maas-api-openshift-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:14:54.731016 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:54.730990    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0818834a-f7a8-4a39-97c4-159253f365df-maas-api-xks-ca" pod "0818834a-f7a8-4a39-97c4-159253f365df" (UID: "0818834a-f7a8-4a39-97c4-159253f365df"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:14:54.731108 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:54.731051    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0818834a-f7a8-4a39-97c4-159253f365df-kube-api-access-6bmmh" pod "0818834a-f7a8-4a39-97c4-159253f365df" (UID: "0818834a-f7a8-4a39-97c4-159253f365df"). InnerVolumeSpecName "kube-api-access-6bmmh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:14:54.831083 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:54.831050    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0818834a-f7a8-4a39-97c4-159253f365df-maas-api-xks-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:14:54.831083 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:54.831084    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bmmh\" (UniqueName: \"kubernetes.io/projected/0818834a-f7a8-4a39-97c4-159253f365df-kube-api-access-6bmmh\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:14:55.442194 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:55.442163    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53f4fd0e-rwrt9"
Oct 30 22:14:55.442362 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:55.442167    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53f4fd0e-rwrt9" event={"ID":"0818834a-f7a8-4a39-97c4-159253f365df","Type":"ContainerDied","Data":"6eb68382fdb86c7c8dc9aaeb27ff28b703f8bae02ca719dbc3d70124f0d86a32"}
Oct 30 22:14:55.442362 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:14:55.442287    5678 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6eb68382fdb86c7c8dc9aaeb27ff28b703f8bae02ca719dbc3d70124f0d86a32"
Oct 30 22:15:00.192944 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:00.192904    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a00d9b-29889975-7nlt8"]
Oct 30 22:15:00.193575 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:00.193557    5678 cpu_manager.go:414] "containerMap: removing container" podUID="0818834a-f7a8-4a39-97c4-159253f365df" containerName="revoke-keys"
Oct 30 22:15:00.193575 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:00.193574    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0818834a-f7a8-4a39-97c4-159253f365df" containerName="revoke-keys"
Oct 30 22:15:00.193731 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:00.193672    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="0818834a-f7a8-4a39-97c4-159253f365df" containerName="revoke-keys"
Oct 30 22:15:00.197430 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:00.197394    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a00d9b-29889975-7nlt8"
Oct 30 22:15:00.218613 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:00.218585    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9dbe6-29889975-x9bnd"]
Oct 30 22:15:00.223083 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:00.223061    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9dbe6-29889975-x9bnd"
Oct 30 22:15:00.227350 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:00.227322    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb0b93-29889975-clfmz"]
Oct 30 22:15:00.228092 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:00.228069    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zz5wr\" (UniqueName: \"kubernetes.io/projected/569c8343-cc6a-437e-b488-0eabfac00d04-kube-api-access-zz5wr\") pod \"maas-api-key-cleanup-e2e-worker-w1-a00d9b-29889975-7nlt8\" (UID: \"569c8343-cc6a-437e-b488-0eabfac00d04\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a00d9b-29889975-7nlt8"
Oct 30 22:15:00.228212 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:00.228195    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/569c8343-cc6a-437e-b488-0eabfac00d04-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-a00d9b-29889975-7nlt8\" (UID: \"569c8343-cc6a-437e-b488-0eabfac00d04\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a00d9b-29889975-7nlt8"
Oct 30 22:15:00.232430 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:00.232378    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb0b93-29889975-clfmz"
Oct 30 22:15:00.243040 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:00.243015    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-779f32-29889975-jb2qb"]
Oct 30 22:15:00.247337 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:00.247315    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-779f32-29889975-jb2qb"
Oct 30 22:15:00.252254 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:00.252209    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5rvcj\""
Oct 30 22:15:00.267587 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:00.267557    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a00d9b-29889975-7nlt8"]
Oct 30 22:15:00.313995 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:00.313262    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9dbe6-29889975-x9bnd"]
Oct 30 22:15:00.320948 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:00.320917    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-779f32-29889975-jb2qb"]
Oct 30 22:15:00.329458 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:00.329423    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gklz7\" (UniqueName: \"kubernetes.io/projected/6c4cd55b-361c-49a8-9ea0-2bb45f97c28d-kube-api-access-gklz7\") pod \"maas-api-key-cleanup-e2e-worker-w6-fb0b93-29889975-clfmz\" (UID: \"6c4cd55b-361c-49a8-9ea0-2bb45f97c28d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb0b93-29889975-clfmz"
Oct 30 22:15:00.329576 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:00.329532    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/569c8343-cc6a-437e-b488-0eabfac00d04-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-a00d9b-29889975-7nlt8\" (UID: \"569c8343-cc6a-437e-b488-0eabfac00d04\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a00d9b-29889975-7nlt8"
Oct 30 22:15:00.329628 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:00.329592    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a104de2-358d-47b3-b066-b8799c301c5c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-c9dbe6-29889975-x9bnd\" (UID: \"6a104de2-358d-47b3-b066-b8799c301c5c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9dbe6-29889975-x9bnd"
Oct 30 22:15:00.329670 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:00.329647    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c4cd55b-361c-49a8-9ea0-2bb45f97c28d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-fb0b93-29889975-clfmz\" (UID: \"6c4cd55b-361c-49a8-9ea0-2bb45f97c28d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb0b93-29889975-clfmz"
Oct 30 22:15:00.329791 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:00.329767    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zsvw\" (UniqueName: \"kubernetes.io/projected/3fc7905d-2cf4-4e42-a606-29ddd24529e1-kube-api-access-8zsvw\") pod \"maas-api-key-cleanup-e2e-worker-w0-779f32-29889975-jb2qb\" (UID: \"3fc7905d-2cf4-4e42-a606-29ddd24529e1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-779f32-29889975-jb2qb"
Oct 30 22:15:00.329884 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:00.329870    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkcfj\" (UniqueName: \"kubernetes.io/projected/6a104de2-358d-47b3-b066-b8799c301c5c-kube-api-access-qkcfj\") pod \"maas-api-key-cleanup-e2e-worker-w5-c9dbe6-29889975-x9bnd\" (UID: \"6a104de2-358d-47b3-b066-b8799c301c5c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9dbe6-29889975-x9bnd"
Oct 30 22:15:00.329968 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:00.329953    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fc7905d-2cf4-4e42-a606-29ddd24529e1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-779f32-29889975-jb2qb\" (UID: \"3fc7905d-2cf4-4e42-a606-29ddd24529e1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-779f32-29889975-jb2qb"
Oct 30 22:15:00.330015 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:00.330006    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zz5wr\" (UniqueName: \"kubernetes.io/projected/569c8343-cc6a-437e-b488-0eabfac00d04-kube-api-access-zz5wr\") pod \"maas-api-key-cleanup-e2e-worker-w1-a00d9b-29889975-7nlt8\" (UID: \"569c8343-cc6a-437e-b488-0eabfac00d04\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a00d9b-29889975-7nlt8"
Oct 30 22:15:00.330351 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:00.330325    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/569c8343-cc6a-437e-b488-0eabfac00d04-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-a00d9b-29889975-7nlt8\" (UID: \"569c8343-cc6a-437e-b488-0eabfac00d04\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a00d9b-29889975-7nlt8"
Oct 30 22:15:00.340147 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:00.340120    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb0b93-29889975-clfmz"]
Oct 30 22:15:00.390933 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:00.390909    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zz5wr\" (UniqueName: \"kubernetes.io/projected/569c8343-cc6a-437e-b488-0eabfac00d04-kube-api-access-zz5wr\") pod \"maas-api-key-cleanup-e2e-worker-w1-a00d9b-29889975-7nlt8\" (UID: \"569c8343-cc6a-437e-b488-0eabfac00d04\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a00d9b-29889975-7nlt8"
Oct 30 22:15:00.431671 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:00.431638    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a104de2-358d-47b3-b066-b8799c301c5c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-c9dbe6-29889975-x9bnd\" (UID: \"6a104de2-358d-47b3-b066-b8799c301c5c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9dbe6-29889975-x9bnd"
Oct 30 22:15:00.431853 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:00.431712    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c4cd55b-361c-49a8-9ea0-2bb45f97c28d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-fb0b93-29889975-clfmz\" (UID: \"6c4cd55b-361c-49a8-9ea0-2bb45f97c28d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb0b93-29889975-clfmz"
Oct 30 22:15:00.431853 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:00.431777    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zsvw\" (UniqueName: \"kubernetes.io/projected/3fc7905d-2cf4-4e42-a606-29ddd24529e1-kube-api-access-8zsvw\") pod \"maas-api-key-cleanup-e2e-worker-w0-779f32-29889975-jb2qb\" (UID: \"3fc7905d-2cf4-4e42-a606-29ddd24529e1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-779f32-29889975-jb2qb"
Oct 30 22:15:00.431853 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:00.431836    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkcfj\" (UniqueName: \"kubernetes.io/projected/6a104de2-358d-47b3-b066-b8799c301c5c-kube-api-access-qkcfj\") pod \"maas-api-key-cleanup-e2e-worker-w5-c9dbe6-29889975-x9bnd\" (UID: \"6a104de2-358d-47b3-b066-b8799c301c5c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9dbe6-29889975-x9bnd"
Oct 30 22:15:00.432035 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:00.431905    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fc7905d-2cf4-4e42-a606-29ddd24529e1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-779f32-29889975-jb2qb\" (UID: \"3fc7905d-2cf4-4e42-a606-29ddd24529e1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-779f32-29889975-jb2qb"
Oct 30 22:15:00.432035 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:00.431995    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gklz7\" (UniqueName: \"kubernetes.io/projected/6c4cd55b-361c-49a8-9ea0-2bb45f97c28d-kube-api-access-gklz7\") pod \"maas-api-key-cleanup-e2e-worker-w6-fb0b93-29889975-clfmz\" (UID: \"6c4cd55b-361c-49a8-9ea0-2bb45f97c28d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb0b93-29889975-clfmz"
Oct 30 22:15:00.432455 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:00.432388    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a104de2-358d-47b3-b066-b8799c301c5c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-c9dbe6-29889975-x9bnd\" (UID: \"6a104de2-358d-47b3-b066-b8799c301c5c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9dbe6-29889975-x9bnd"
Oct 30 22:15:00.432516 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:00.432472    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c4cd55b-361c-49a8-9ea0-2bb45f97c28d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-fb0b93-29889975-clfmz\" (UID: \"6c4cd55b-361c-49a8-9ea0-2bb45f97c28d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb0b93-29889975-clfmz"
Oct 30 22:15:00.432727 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:00.432711    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fc7905d-2cf4-4e42-a606-29ddd24529e1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-779f32-29889975-jb2qb\" (UID: \"3fc7905d-2cf4-4e42-a606-29ddd24529e1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-779f32-29889975-jb2qb"
Oct 30 22:15:00.492249 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:00.492185    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkcfj\" (UniqueName: \"kubernetes.io/projected/6a104de2-358d-47b3-b066-b8799c301c5c-kube-api-access-qkcfj\") pod \"maas-api-key-cleanup-e2e-worker-w5-c9dbe6-29889975-x9bnd\" (UID: \"6a104de2-358d-47b3-b066-b8799c301c5c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9dbe6-29889975-x9bnd"
Oct 30 22:15:00.492561 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:00.492535    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gklz7\" (UniqueName: \"kubernetes.io/projected/6c4cd55b-361c-49a8-9ea0-2bb45f97c28d-kube-api-access-gklz7\") pod \"maas-api-key-cleanup-e2e-worker-w6-fb0b93-29889975-clfmz\" (UID: \"6c4cd55b-361c-49a8-9ea0-2bb45f97c28d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb0b93-29889975-clfmz"
Oct 30 22:15:00.493196 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:00.493173    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zsvw\" (UniqueName: \"kubernetes.io/projected/3fc7905d-2cf4-4e42-a606-29ddd24529e1-kube-api-access-8zsvw\") pod \"maas-api-key-cleanup-e2e-worker-w0-779f32-29889975-jb2qb\" (UID: \"3fc7905d-2cf4-4e42-a606-29ddd24529e1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-779f32-29889975-jb2qb"
Oct 30 22:15:00.513665 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:00.513639    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a00d9b-29889975-7nlt8"
Oct 30 22:15:00.538234 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:00.538210    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9dbe6-29889975-x9bnd"
Oct 30 22:15:00.548546 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:00.548524    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb0b93-29889975-clfmz"
Oct 30 22:15:00.562801 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:00.562774    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-779f32-29889975-jb2qb"
Oct 30 22:15:00.918768 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:00.918738    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a00d9b-29889975-7nlt8"]
Oct 30 22:15:00.919584 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:15:00.919539    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod569c8343_cc6a_437e_b488_0eabfac00d04.slice/crio-d9851305ac5891e51fc45d030f7208a500e2a8d770333ee7dfc404785fcd19e5 WatchSource:0}: Error finding container d9851305ac5891e51fc45d030f7208a500e2a8d770333ee7dfc404785fcd19e5: Status 404 returned error can't find the container with id d9851305ac5891e51fc45d030f7208a500e2a8d770333ee7dfc404785fcd19e5
Oct 30 22:15:00.966748 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:15:00.966719    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a104de2_358d_47b3_b066_b8799c301c5c.slice/crio-6a4a44823e218a80069a91d11c54e03a42327cc0f8432c429d238ba37a5bbcbf WatchSource:0}: Error finding container 6a4a44823e218a80069a91d11c54e03a42327cc0f8432c429d238ba37a5bbcbf: Status 404 returned error can't find the container with id 6a4a44823e218a80069a91d11c54e03a42327cc0f8432c429d238ba37a5bbcbf
Oct 30 22:15:00.967893 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:00.967866    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9dbe6-29889975-x9bnd"]
Oct 30 22:15:01.018588 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:15:01.018560    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c4cd55b_361c_49a8_9ea0_2bb45f97c28d.slice/crio-427485d83151453a0b891c25e048c2c0d051da9e68e4004f582233ad6ab89a74 WatchSource:0}: Error finding container 427485d83151453a0b891c25e048c2c0d051da9e68e4004f582233ad6ab89a74: Status 404 returned error can't find the container with id 427485d83151453a0b891c25e048c2c0d051da9e68e4004f582233ad6ab89a74
Oct 30 22:15:01.020266 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:15:01.020236    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fc7905d_2cf4_4e42_a606_29ddd24529e1.slice/crio-66741602c593f117804e1713b6a4f34fccdf2be759a8c23cb4bc1de9a3252c29 WatchSource:0}: Error finding container 66741602c593f117804e1713b6a4f34fccdf2be759a8c23cb4bc1de9a3252c29: Status 404 returned error can't find the container with id 66741602c593f117804e1713b6a4f34fccdf2be759a8c23cb4bc1de9a3252c29
Oct 30 22:15:01.021850 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:01.021776    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-779f32-29889975-jb2qb"]
Oct 30 22:15:01.027247 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:01.027225    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb0b93-29889975-clfmz"]
Oct 30 22:15:01.531154 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:01.531123    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a00d9b-29889975-7nlt8" event={"ID":"569c8343-cc6a-437e-b488-0eabfac00d04","Type":"ContainerStarted","Data":"831ab6df2e1cda650640e6108f81e9f411d13e1212e14bcd3c8a93c0610bb209"}
Oct 30 22:15:01.531800 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:01.531183    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a00d9b-29889975-7nlt8" event={"ID":"569c8343-cc6a-437e-b488-0eabfac00d04","Type":"ContainerStarted","Data":"d9851305ac5891e51fc45d030f7208a500e2a8d770333ee7dfc404785fcd19e5"}
Oct 30 22:15:01.533080 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:01.533055    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-779f32-29889975-jb2qb" event={"ID":"3fc7905d-2cf4-4e42-a606-29ddd24529e1","Type":"ContainerStarted","Data":"e60c4b0cb76a5d924d2928d0725625e729f77c6ac6cdba39466fe7431dbe9ed6"}
Oct 30 22:15:01.533182 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:01.533104    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-779f32-29889975-jb2qb" event={"ID":"3fc7905d-2cf4-4e42-a606-29ddd24529e1","Type":"ContainerStarted","Data":"66741602c593f117804e1713b6a4f34fccdf2be759a8c23cb4bc1de9a3252c29"}
Oct 30 22:15:01.534723 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:01.534695    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb0b93-29889975-clfmz" event={"ID":"6c4cd55b-361c-49a8-9ea0-2bb45f97c28d","Type":"ContainerStarted","Data":"427485d83151453a0b891c25e048c2c0d051da9e68e4004f582233ad6ab89a74"}
Oct 30 22:15:01.536201 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:01.536182    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9dbe6-29889975-x9bnd" event={"ID":"6a104de2-358d-47b3-b066-b8799c301c5c","Type":"ContainerStarted","Data":"018a37675e4288b67f20b6654cbf0bdf0492bf40eda1366e76cdc3affbb8ebf1"}
Oct 30 22:15:01.536296 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:01.536216    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9dbe6-29889975-x9bnd" event={"ID":"6a104de2-358d-47b3-b066-b8799c301c5c","Type":"ContainerStarted","Data":"6a4a44823e218a80069a91d11c54e03a42327cc0f8432c429d238ba37a5bbcbf"}
Oct 30 22:15:01.646862 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:01.646811    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a00d9b-29889975-7nlt8" podStartSLOduration=1.646797099 podStartE2EDuration="1.646797099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:01.643564964 +0000 UTC m=+1479.247872530" watchObservedRunningTime="2026-10-30 22:15:01.646797099 +0000 UTC m=+1479.251104652"
Oct 30 22:15:01.778387 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:01.778334    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9dbe6-29889975-x9bnd" podStartSLOduration=1.778320591 podStartE2EDuration="1.778320591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:01.776462351 +0000 UTC m=+1479.380769903" watchObservedRunningTime="2026-10-30 22:15:01.778320591 +0000 UTC m=+1479.382628143"
Oct 30 22:15:01.929335 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:01.929268    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-779f32-29889975-jb2qb" podStartSLOduration=1.929248062 podStartE2EDuration="1.929248062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:01.924784453 +0000 UTC m=+1479.529092030" watchObservedRunningTime="2026-10-30 22:15:01.929248062 +0000 UTC m=+1479.533555617"
Oct 30 22:15:02.546290 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:02.546259    5678 generic.go:356] "Generic (PLEG): container finished" podID="3fc7905d-2cf4-4e42-a606-29ddd24529e1" containerID="e60c4b0cb76a5d924d2928d0725625e729f77c6ac6cdba39466fe7431dbe9ed6" exitCode=60
Oct 30 22:15:02.546807 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:02.546346    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-779f32-29889975-jb2qb" event={"ID":"3fc7905d-2cf4-4e42-a606-29ddd24529e1","Type":"ContainerDied","Data":"e60c4b0cb76a5d924d2928d0725625e729f77c6ac6cdba39466fe7431dbe9ed6"}
Oct 30 22:15:02.546807 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:02.546624    5678 scope.go:122] "RemoveContainer" containerID="e60c4b0cb76a5d924d2928d0725625e729f77c6ac6cdba39466fe7431dbe9ed6"
Oct 30 22:15:02.548111 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:02.548088    5678 generic.go:356] "Generic (PLEG): container finished" podID="6c4cd55b-361c-49a8-9ea0-2bb45f97c28d" containerID="255e6e974e07a8aad0f9e47734e47f16ef03fd073cfc3594d185e1df614bf1a7" exitCode=60
Oct 30 22:15:02.548218 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:02.548158    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb0b93-29889975-clfmz" event={"ID":"6c4cd55b-361c-49a8-9ea0-2bb45f97c28d","Type":"ContainerDied","Data":"255e6e974e07a8aad0f9e47734e47f16ef03fd073cfc3594d185e1df614bf1a7"}
Oct 30 22:15:02.548494 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:02.548476    5678 scope.go:122] "RemoveContainer" containerID="255e6e974e07a8aad0f9e47734e47f16ef03fd073cfc3594d185e1df614bf1a7"
Oct 30 22:15:02.555590 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:02.555571    5678 generic.go:356] "Generic (PLEG): container finished" podID="6a104de2-358d-47b3-b066-b8799c301c5c" containerID="018a37675e4288b67f20b6654cbf0bdf0492bf40eda1366e76cdc3affbb8ebf1" exitCode=60
Oct 30 22:15:02.555693 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:02.555672    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9dbe6-29889975-x9bnd" event={"ID":"6a104de2-358d-47b3-b066-b8799c301c5c","Type":"ContainerDied","Data":"018a37675e4288b67f20b6654cbf0bdf0492bf40eda1366e76cdc3affbb8ebf1"}
Oct 30 22:15:02.555899 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:02.555830    5678 scope.go:122] "RemoveContainer" containerID="018a37675e4288b67f20b6654cbf0bdf0492bf40eda1366e76cdc3affbb8ebf1"
Oct 30 22:15:02.563000 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:02.558020    5678 generic.go:356] "Generic (PLEG): container finished" podID="569c8343-cc6a-437e-b488-0eabfac00d04" containerID="831ab6df2e1cda650640e6108f81e9f411d13e1212e14bcd3c8a93c0610bb209" exitCode=60
Oct 30 22:15:02.563000 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:02.558202    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a00d9b-29889975-7nlt8" event={"ID":"569c8343-cc6a-437e-b488-0eabfac00d04","Type":"ContainerDied","Data":"831ab6df2e1cda650640e6108f81e9f411d13e1212e14bcd3c8a93c0610bb209"}
Oct 30 22:15:02.563000 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:02.558356    5678 scope.go:122] "RemoveContainer" containerID="831ab6df2e1cda650640e6108f81e9f411d13e1212e14bcd3c8a93c0610bb209"
Oct 30 22:15:03.568453 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:03.568344    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a00d9b-29889975-7nlt8" event={"ID":"569c8343-cc6a-437e-b488-0eabfac00d04","Type":"ContainerStarted","Data":"04af45220d43d287322cf4dc1807676eb59a4db8296b68660c5957e417f8a045"}
Oct 30 22:15:03.570480 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:03.570448    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-779f32-29889975-jb2qb" event={"ID":"3fc7905d-2cf4-4e42-a606-29ddd24529e1","Type":"ContainerStarted","Data":"3e4e2745a388848fc93877f346ac62fda7b332ae14e092271244fe26fe4df5bd"}
Oct 30 22:15:03.572262 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:03.572234    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb0b93-29889975-clfmz" event={"ID":"6c4cd55b-361c-49a8-9ea0-2bb45f97c28d","Type":"ContainerStarted","Data":"df7b006bc2f4cf441807a71b98030b016f6ad35ead7c4f3cf6c91f461c495308"}
Oct 30 22:15:03.574053 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:03.574034    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9dbe6-29889975-x9bnd" event={"ID":"6a104de2-358d-47b3-b066-b8799c301c5c","Type":"ContainerStarted","Data":"fb76ad9d82fb54d6c410614bfd81693b82c356f940a74cff79416971422949eb"}
Oct 30 22:15:03.905587 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:03.905508    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb0b93-29889975-clfmz" podStartSLOduration=3.905491393 podStartE2EDuration="3.905491393s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:03.903493734 +0000 UTC m=+1481.507801287" watchObservedRunningTime="2026-10-30 22:15:03.905491393 +0000 UTC m=+1481.509798977"
Oct 30 22:15:04.600106 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:04.600071    5678 generic.go:356] "Generic (PLEG): container finished" podID="3fc7905d-2cf4-4e42-a606-29ddd24529e1" containerID="3e4e2745a388848fc93877f346ac62fda7b332ae14e092271244fe26fe4df5bd" exitCode=60
Oct 30 22:15:04.600621 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:04.600186    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-779f32-29889975-jb2qb" event={"ID":"3fc7905d-2cf4-4e42-a606-29ddd24529e1","Type":"ContainerDied","Data":"3e4e2745a388848fc93877f346ac62fda7b332ae14e092271244fe26fe4df5bd"}
Oct 30 22:15:04.600621 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:04.600247    5678 scope.go:122] "RemoveContainer" containerID="e60c4b0cb76a5d924d2928d0725625e729f77c6ac6cdba39466fe7431dbe9ed6"
Oct 30 22:15:04.600862 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:04.600836    5678 scope.go:122] "RemoveContainer" containerID="3e4e2745a388848fc93877f346ac62fda7b332ae14e092271244fe26fe4df5bd"
Oct 30 22:15:04.604801 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:04.604771    5678 generic.go:356] "Generic (PLEG): container finished" podID="6c4cd55b-361c-49a8-9ea0-2bb45f97c28d" containerID="df7b006bc2f4cf441807a71b98030b016f6ad35ead7c4f3cf6c91f461c495308" exitCode=60
Oct 30 22:15:04.604915 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:04.604865    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb0b93-29889975-clfmz" event={"ID":"6c4cd55b-361c-49a8-9ea0-2bb45f97c28d","Type":"ContainerDied","Data":"df7b006bc2f4cf441807a71b98030b016f6ad35ead7c4f3cf6c91f461c495308"}
Oct 30 22:15:04.606492 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:15:04.606454    5678 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-779f32-29889975-jb2qb_odh-ai-gateway-infra(3fc7905d-2cf4-4e42-a606-29ddd24529e1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-779f32-29889975-jb2qb" podUID="3fc7905d-2cf4-4e42-a606-29ddd24529e1"
Oct 30 22:15:04.606697 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:04.606681    5678 scope.go:122] "RemoveContainer" containerID="df7b006bc2f4cf441807a71b98030b016f6ad35ead7c4f3cf6c91f461c495308"
Oct 30 22:15:04.607002 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:15:04.606981    5678 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-fb0b93-29889975-clfmz_odh-ai-gateway-infra(6c4cd55b-361c-49a8-9ea0-2bb45f97c28d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb0b93-29889975-clfmz" podUID="6c4cd55b-361c-49a8-9ea0-2bb45f97c28d"
Oct 30 22:15:04.607748 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:04.607725    5678 generic.go:356] "Generic (PLEG): container finished" podID="6a104de2-358d-47b3-b066-b8799c301c5c" containerID="fb76ad9d82fb54d6c410614bfd81693b82c356f940a74cff79416971422949eb" exitCode=60
Oct 30 22:15:04.607891 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:04.607805    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9dbe6-29889975-x9bnd" event={"ID":"6a104de2-358d-47b3-b066-b8799c301c5c","Type":"ContainerDied","Data":"fb76ad9d82fb54d6c410614bfd81693b82c356f940a74cff79416971422949eb"}
Oct 30 22:15:04.608052 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:04.608032    5678 scope.go:122] "RemoveContainer" containerID="fb76ad9d82fb54d6c410614bfd81693b82c356f940a74cff79416971422949eb"
Oct 30 22:15:04.608361 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:15:04.608338    5678 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-c9dbe6-29889975-x9bnd_odh-ai-gateway-infra(6a104de2-358d-47b3-b066-b8799c301c5c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9dbe6-29889975-x9bnd" podUID="6a104de2-358d-47b3-b066-b8799c301c5c"
Oct 30 22:15:04.609958 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:04.609931    5678 generic.go:356] "Generic (PLEG): container finished" podID="569c8343-cc6a-437e-b488-0eabfac00d04" containerID="04af45220d43d287322cf4dc1807676eb59a4db8296b68660c5957e417f8a045" exitCode=60
Oct 30 22:15:04.610050 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:04.610006    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a00d9b-29889975-7nlt8" event={"ID":"569c8343-cc6a-437e-b488-0eabfac00d04","Type":"ContainerDied","Data":"04af45220d43d287322cf4dc1807676eb59a4db8296b68660c5957e417f8a045"}
Oct 30 22:15:04.610181 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:04.610166    5678 scope.go:122] "RemoveContainer" containerID="04af45220d43d287322cf4dc1807676eb59a4db8296b68660c5957e417f8a045"
Oct 30 22:15:04.610442 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:15:04.610378    5678 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-a00d9b-29889975-7nlt8_odh-ai-gateway-infra(569c8343-cc6a-437e-b488-0eabfac00d04)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a00d9b-29889975-7nlt8" podUID="569c8343-cc6a-437e-b488-0eabfac00d04"
Oct 30 22:15:04.620164 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:04.620137    5678 scope.go:122] "RemoveContainer" containerID="255e6e974e07a8aad0f9e47734e47f16ef03fd073cfc3594d185e1df614bf1a7"
Oct 30 22:15:04.636445 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:04.636421    5678 scope.go:122] "RemoveContainer" containerID="018a37675e4288b67f20b6654cbf0bdf0492bf40eda1366e76cdc3affbb8ebf1"
Oct 30 22:15:04.657545 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:04.657517    5678 scope.go:122] "RemoveContainer" containerID="831ab6df2e1cda650640e6108f81e9f411d13e1212e14bcd3c8a93c0610bb209"
Oct 30 22:15:05.682422 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:05.679009    5678 scope.go:122] "RemoveContainer" containerID="3e4e2745a388848fc93877f346ac62fda7b332ae14e092271244fe26fe4df5bd"
Oct 30 22:15:05.682422 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:15:05.679469    5678 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-779f32-29889975-jb2qb_odh-ai-gateway-infra(3fc7905d-2cf4-4e42-a606-29ddd24529e1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-779f32-29889975-jb2qb" podUID="3fc7905d-2cf4-4e42-a606-29ddd24529e1"
Oct 30 22:15:05.683657 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:05.683633    5678 scope.go:122] "RemoveContainer" containerID="df7b006bc2f4cf441807a71b98030b016f6ad35ead7c4f3cf6c91f461c495308"
Oct 30 22:15:05.684001 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:15:05.683976    5678 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-fb0b93-29889975-clfmz_odh-ai-gateway-infra(6c4cd55b-361c-49a8-9ea0-2bb45f97c28d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb0b93-29889975-clfmz" podUID="6c4cd55b-361c-49a8-9ea0-2bb45f97c28d"
Oct 30 22:15:05.686819 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:05.686798    5678 scope.go:122] "RemoveContainer" containerID="fb76ad9d82fb54d6c410614bfd81693b82c356f940a74cff79416971422949eb"
Oct 30 22:15:05.687124 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:15:05.687100    5678 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-c9dbe6-29889975-x9bnd_odh-ai-gateway-infra(6a104de2-358d-47b3-b066-b8799c301c5c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9dbe6-29889975-x9bnd" podUID="6a104de2-358d-47b3-b066-b8799c301c5c"
Oct 30 22:15:05.690132 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:05.690115    5678 scope.go:122] "RemoveContainer" containerID="04af45220d43d287322cf4dc1807676eb59a4db8296b68660c5957e417f8a045"
Oct 30 22:15:05.690332 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:15:05.690313    5678 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-a00d9b-29889975-7nlt8_odh-ai-gateway-infra(569c8343-cc6a-437e-b488-0eabfac00d04)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a00d9b-29889975-7nlt8" podUID="569c8343-cc6a-437e-b488-0eabfac00d04"
Oct 30 22:15:06.526885 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:06.526852    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-ff716ef6-openshift-default-5bbc788795-f68kp"]
Oct 30 22:15:06.533047 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:06.533022    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ff716ef6-openshift-default-5bbc788795-f68kp"
Oct 30 22:15:06.561264 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:06.561133    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/313acbe0-dc5c-48cb-92d8-972092227317-istio-data\") pod \"e2e-ait-ff716ef6-openshift-default-5bbc788795-f68kp\" (UID: \"313acbe0-dc5c-48cb-92d8-972092227317\") " pod="openshift-ingress/e2e-ait-ff716ef6-openshift-default-5bbc788795-f68kp"
Oct 30 22:15:06.561264 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:06.561178    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/313acbe0-dc5c-48cb-92d8-972092227317-istiod-ca-cert\") pod \"e2e-ait-ff716ef6-openshift-default-5bbc788795-f68kp\" (UID: \"313acbe0-dc5c-48cb-92d8-972092227317\") " pod="openshift-ingress/e2e-ait-ff716ef6-openshift-default-5bbc788795-f68kp"
Oct 30 22:15:06.561491 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:06.561421    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/313acbe0-dc5c-48cb-92d8-972092227317-istio-token\") pod \"e2e-ait-ff716ef6-openshift-default-5bbc788795-f68kp\" (UID: \"313acbe0-dc5c-48cb-92d8-972092227317\") " pod="openshift-ingress/e2e-ait-ff716ef6-openshift-default-5bbc788795-f68kp"
Oct 30 22:15:06.561563 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:06.561529    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f42nb\" (UniqueName: \"kubernetes.io/projected/313acbe0-dc5c-48cb-92d8-972092227317-kube-api-access-f42nb\") pod \"e2e-ait-ff716ef6-openshift-default-5bbc788795-f68kp\" (UID: \"313acbe0-dc5c-48cb-92d8-972092227317\") " pod="openshift-ingress/e2e-ait-ff716ef6-openshift-default-5bbc788795-f68kp"
Oct 30 22:15:06.561734 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:06.561705    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/313acbe0-dc5c-48cb-92d8-972092227317-workload-certs\") pod \"e2e-ait-ff716ef6-openshift-default-5bbc788795-f68kp\" (UID: \"313acbe0-dc5c-48cb-92d8-972092227317\") " pod="openshift-ingress/e2e-ait-ff716ef6-openshift-default-5bbc788795-f68kp"
Oct 30 22:15:06.561802 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:06.561780    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/313acbe0-dc5c-48cb-92d8-972092227317-istio-envoy\") pod \"e2e-ait-ff716ef6-openshift-default-5bbc788795-f68kp\" (UID: \"313acbe0-dc5c-48cb-92d8-972092227317\") " pod="openshift-ingress/e2e-ait-ff716ef6-openshift-default-5bbc788795-f68kp"
Oct 30 22:15:06.561864 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:06.561851    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/313acbe0-dc5c-48cb-92d8-972092227317-workload-socket\") pod \"e2e-ait-ff716ef6-openshift-default-5bbc788795-f68kp\" (UID: \"313acbe0-dc5c-48cb-92d8-972092227317\") " pod="openshift-ingress/e2e-ait-ff716ef6-openshift-default-5bbc788795-f68kp"
Oct 30 22:15:06.562012 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:06.561994    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/313acbe0-dc5c-48cb-92d8-972092227317-credential-socket\") pod \"e2e-ait-ff716ef6-openshift-default-5bbc788795-f68kp\" (UID: \"313acbe0-dc5c-48cb-92d8-972092227317\") " pod="openshift-ingress/e2e-ait-ff716ef6-openshift-default-5bbc788795-f68kp"
Oct 30 22:15:06.562078 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:06.562025    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/313acbe0-dc5c-48cb-92d8-972092227317-istio-podinfo\") pod \"e2e-ait-ff716ef6-openshift-default-5bbc788795-f68kp\" (UID: \"313acbe0-dc5c-48cb-92d8-972092227317\") " pod="openshift-ingress/e2e-ait-ff716ef6-openshift-default-5bbc788795-f68kp"
Oct 30 22:15:06.588199 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:06.588163    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-ff716ef6-openshift-default-5bbc788795-f68kp"]
Oct 30 22:15:06.627334 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:06.627297    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-ff716ef6-openshift-default-dockercfg-66b79\""
Oct 30 22:15:06.663069 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:06.663035    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f42nb\" (UniqueName: \"kubernetes.io/projected/313acbe0-dc5c-48cb-92d8-972092227317-kube-api-access-f42nb\") pod \"e2e-ait-ff716ef6-openshift-default-5bbc788795-f68kp\" (UID: \"313acbe0-dc5c-48cb-92d8-972092227317\") " pod="openshift-ingress/e2e-ait-ff716ef6-openshift-default-5bbc788795-f68kp"
Oct 30 22:15:06.663069 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:06.663134    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/313acbe0-dc5c-48cb-92d8-972092227317-workload-certs\") pod \"e2e-ait-ff716ef6-openshift-default-5bbc788795-f68kp\" (UID: \"313acbe0-dc5c-48cb-92d8-972092227317\") " pod="openshift-ingress/e2e-ait-ff716ef6-openshift-default-5bbc788795-f68kp"
Oct 30 22:15:06.663069 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:06.663160    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/313acbe0-dc5c-48cb-92d8-972092227317-istio-envoy\") pod \"e2e-ait-ff716ef6-openshift-default-5bbc788795-f68kp\" (UID: \"313acbe0-dc5c-48cb-92d8-972092227317\") " pod="openshift-ingress/e2e-ait-ff716ef6-openshift-default-5bbc788795-f68kp"
Oct 30 22:15:06.663890 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:06.663449    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/313acbe0-dc5c-48cb-92d8-972092227317-workload-socket\") pod \"e2e-ait-ff716ef6-openshift-default-5bbc788795-f68kp\" (UID: \"313acbe0-dc5c-48cb-92d8-972092227317\") " pod="openshift-ingress/e2e-ait-ff716ef6-openshift-default-5bbc788795-f68kp"
Oct 30 22:15:06.663890 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:06.663863    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/313acbe0-dc5c-48cb-92d8-972092227317-workload-certs\") pod \"e2e-ait-ff716ef6-openshift-default-5bbc788795-f68kp\" (UID: \"313acbe0-dc5c-48cb-92d8-972092227317\") " pod="openshift-ingress/e2e-ait-ff716ef6-openshift-default-5bbc788795-f68kp"
Oct 30 22:15:06.664112 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:06.663983    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/313acbe0-dc5c-48cb-92d8-972092227317-credential-socket\") pod \"e2e-ait-ff716ef6-openshift-default-5bbc788795-f68kp\" (UID: \"313acbe0-dc5c-48cb-92d8-972092227317\") " pod="openshift-ingress/e2e-ait-ff716ef6-openshift-default-5bbc788795-f68kp"
Oct 30 22:15:06.664112 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:06.664034    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/313acbe0-dc5c-48cb-92d8-972092227317-istio-podinfo\") pod \"e2e-ait-ff716ef6-openshift-default-5bbc788795-f68kp\" (UID: \"313acbe0-dc5c-48cb-92d8-972092227317\") " pod="openshift-ingress/e2e-ait-ff716ef6-openshift-default-5bbc788795-f68kp"
Oct 30 22:15:06.664112 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:06.664084    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/313acbe0-dc5c-48cb-92d8-972092227317-workload-socket\") pod \"e2e-ait-ff716ef6-openshift-default-5bbc788795-f68kp\" (UID: \"313acbe0-dc5c-48cb-92d8-972092227317\") " pod="openshift-ingress/e2e-ait-ff716ef6-openshift-default-5bbc788795-f68kp"
Oct 30 22:15:06.664366 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:06.664186    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/313acbe0-dc5c-48cb-92d8-972092227317-istio-data\") pod \"e2e-ait-ff716ef6-openshift-default-5bbc788795-f68kp\" (UID: \"313acbe0-dc5c-48cb-92d8-972092227317\") " pod="openshift-ingress/e2e-ait-ff716ef6-openshift-default-5bbc788795-f68kp"
Oct 30 22:15:06.664366 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:06.664216    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/313acbe0-dc5c-48cb-92d8-972092227317-istiod-ca-cert\") pod \"e2e-ait-ff716ef6-openshift-default-5bbc788795-f68kp\" (UID: \"313acbe0-dc5c-48cb-92d8-972092227317\") " pod="openshift-ingress/e2e-ait-ff716ef6-openshift-default-5bbc788795-f68kp"
Oct 30 22:15:06.664366 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:06.664285    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/313acbe0-dc5c-48cb-92d8-972092227317-istio-token\") pod \"e2e-ait-ff716ef6-openshift-default-5bbc788795-f68kp\" (UID: \"313acbe0-dc5c-48cb-92d8-972092227317\") " pod="openshift-ingress/e2e-ait-ff716ef6-openshift-default-5bbc788795-f68kp"
Oct 30 22:15:06.664760 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:06.664724    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/313acbe0-dc5c-48cb-92d8-972092227317-istio-data\") pod \"e2e-ait-ff716ef6-openshift-default-5bbc788795-f68kp\" (UID: \"313acbe0-dc5c-48cb-92d8-972092227317\") " pod="openshift-ingress/e2e-ait-ff716ef6-openshift-default-5bbc788795-f68kp"
Oct 30 22:15:06.665702 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:06.665676    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/313acbe0-dc5c-48cb-92d8-972092227317-istiod-ca-cert\") pod \"e2e-ait-ff716ef6-openshift-default-5bbc788795-f68kp\" (UID: \"313acbe0-dc5c-48cb-92d8-972092227317\") " pod="openshift-ingress/e2e-ait-ff716ef6-openshift-default-5bbc788795-f68kp"
Oct 30 22:15:06.665972 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:06.665949    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/313acbe0-dc5c-48cb-92d8-972092227317-credential-socket\") pod \"e2e-ait-ff716ef6-openshift-default-5bbc788795-f68kp\" (UID: \"313acbe0-dc5c-48cb-92d8-972092227317\") " pod="openshift-ingress/e2e-ait-ff716ef6-openshift-default-5bbc788795-f68kp"
Oct 30 22:15:06.667742 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:06.667705    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/313acbe0-dc5c-48cb-92d8-972092227317-istio-envoy\") pod \"e2e-ait-ff716ef6-openshift-default-5bbc788795-f68kp\" (UID: \"313acbe0-dc5c-48cb-92d8-972092227317\") " pod="openshift-ingress/e2e-ait-ff716ef6-openshift-default-5bbc788795-f68kp"
Oct 30 22:15:06.670836 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:06.670813    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/313acbe0-dc5c-48cb-92d8-972092227317-istio-podinfo\") pod \"e2e-ait-ff716ef6-openshift-default-5bbc788795-f68kp\" (UID: \"313acbe0-dc5c-48cb-92d8-972092227317\") " pod="openshift-ingress/e2e-ait-ff716ef6-openshift-default-5bbc788795-f68kp"
Oct 30 22:15:06.722271 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:06.722141    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f42nb\" (UniqueName: \"kubernetes.io/projected/313acbe0-dc5c-48cb-92d8-972092227317-kube-api-access-f42nb\") pod \"e2e-ait-ff716ef6-openshift-default-5bbc788795-f68kp\" (UID: \"313acbe0-dc5c-48cb-92d8-972092227317\") " pod="openshift-ingress/e2e-ait-ff716ef6-openshift-default-5bbc788795-f68kp"
Oct 30 22:15:06.723601 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:06.723574    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/313acbe0-dc5c-48cb-92d8-972092227317-istio-token\") pod \"e2e-ait-ff716ef6-openshift-default-5bbc788795-f68kp\" (UID: \"313acbe0-dc5c-48cb-92d8-972092227317\") " pod="openshift-ingress/e2e-ait-ff716ef6-openshift-default-5bbc788795-f68kp"
Oct 30 22:15:06.848036 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:06.847939    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ff716ef6-openshift-default-5bbc788795-f68kp"
Oct 30 22:15:07.309357 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:07.309326    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-ff716ef6-openshift-default-5bbc788795-f68kp"]
Oct 30 22:15:07.339173 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:15:07.339144    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod313acbe0_dc5c_48cb_92d8_972092227317.slice/crio-434d9558dec72c398263b19b0f4e6fbd7e4f8ef43ebd249636a035f804c50a6f WatchSource:0}: Error finding container 434d9558dec72c398263b19b0f4e6fbd7e4f8ef43ebd249636a035f804c50a6f: Status 404 returned error can't find the container with id 434d9558dec72c398263b19b0f4e6fbd7e4f8ef43ebd249636a035f804c50a6f
Oct 30 22:15:07.341859 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:07.341827    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:15:07.341942 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:07.341896    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:15:07.341942 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:07.341925    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:15:07.713475 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:07.713435    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ff716ef6-openshift-default-5bbc788795-f68kp" event={"ID":"313acbe0-dc5c-48cb-92d8-972092227317","Type":"ContainerStarted","Data":"141a472a7e05afdc52b72cd370f0ad46fc2206ebc9a7bae42c951e83738cc0fb"}
Oct 30 22:15:07.713659 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:07.713495    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ff716ef6-openshift-default-5bbc788795-f68kp" event={"ID":"313acbe0-dc5c-48cb-92d8-972092227317","Type":"ContainerStarted","Data":"434d9558dec72c398263b19b0f4e6fbd7e4f8ef43ebd249636a035f804c50a6f"}
Oct 30 22:15:07.833169 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:07.833115    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-ff716ef6-openshift-default-5bbc788795-f68kp" podStartSLOduration=1.833098565 podStartE2EDuration="1.833098565s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:07.830693225 +0000 UTC m=+1485.435000779" watchObservedRunningTime="2026-10-30 22:15:07.833098565 +0000 UTC m=+1485.437406118"
Oct 30 22:15:07.849029 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:07.848998    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-ff716ef6-openshift-default-5bbc788795-f68kp"
Oct 30 22:15:08.854328 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:08.854303    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-ff716ef6-openshift-default-5bbc788795-f68kp"
Oct 30 22:15:09.748937 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:09.743755    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-ff716ef6-openshift-default-5bbc788795-f68kp"
Oct 30 22:15:09.748937 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:09.748619    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-ff716ef6-openshift-default-5bbc788795-f68kp"
Oct 30 22:15:10.691674 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:15:10.691614    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 22:15:12.287427 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:12.281505    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a00d9b/distinct-2-w1-a00d9b-kserve-68fd95957c-dmgnb" podUID="23a81fda-f7e3-4f91-8d96-9f19c7de6e09" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:15:12.326427 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:12.304682    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7bb1c2/simulator-w3-7bb1c2-kserve-675645869c-vcwqv" podUID="1fce9daf-68c9-41d0-825b-2012bb93c8ba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:15:12.326427 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:12.304955    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779f32/premium-w0-779f32-kserve-67978797c8-jpkvl" podUID="40138fb9-9a75-4458-a351-11eb61c4a1e3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Oct 30 22:15:12.326427 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:12.305139    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7bb1c2/distinct-w3-7bb1c2-kserve-8d695596c-8zgtw" podUID="36f0a6a2-54be-49bf-9202-502960cbb134" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded"
Oct 30 22:15:12.548480 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:12.362898    5678 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a00d9b/distinct-2-w1-a00d9b-kserve-68fd95957c-dmgnb" podUID="23a81fda-f7e3-4f91-8d96-9f19c7de6e09" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:15:12.548480 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:12.362953    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-65e59f/simulator-w4-65e59f-kserve-85698bcd47-przch" podUID="ee5e843b-17be-4623-8c95-dc07c8b7f506" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:15:12.548480 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:12.363294    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-65e59f/premium-w4-65e59f-kserve-6dc49468f8-rlzhx" podUID="8590a783-eb24-4759-ad74-1369f36aa9d2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:15:12.548480 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:12.413686    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9ndd9" podUID="81bd6415-f42f-446b-a64e-956f794b3956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": context deadline exceeded"
Oct 30 22:15:12.548480 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:12.414907    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb0b93/simulator-w6-fb0b93-kserve-55b784b479-588bn" podUID="951a6f7e-a22e-40f2-ace2-1ce5d3bc9b49" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:15:12.548480 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:12.419220    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-65e59f/distinct-2-w4-65e59f-kserve-689566757d-bf98q" podUID="64e66fdb-056e-4086-a85a-8c02613584a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:15:12.609672 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:12.609632    5678 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-7bb1c2/distinct-w3-7bb1c2-kserve-8d695596c-8zgtw" podUID="36f0a6a2-54be-49bf-9202-502960cbb134" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:15:12.671949 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:12.671520    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7bb1c2/premium-w3-7bb1c2-kserve-7dbf875d88-dqxn5" podUID="a58c911f-e2eb-4820-a6b6-24afe78b5fd4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:15:12.777470 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:12.777428    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb0b93/distinct-2-w6-fb0b93-kserve-7d65bb4698-m6tkh" podUID="6fd276ef-522b-4bb4-b101-0a9c3fe9ecd3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:15:12.814652 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:12.797454    5678 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-7bb1c2/simulator-w3-7bb1c2-kserve-675645869c-vcwqv" podUID="1fce9daf-68c9-41d0-825b-2012bb93c8ba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:15:13.470432 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:13.469285    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-65e59f/simulator-w4-65e59f-kserve-85698bcd47-przch" podUID="ee5e843b-17be-4623-8c95-dc07c8b7f506" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Oct 30 22:15:13.470432 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:13.469894    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-65e59f/premium-w4-65e59f-kserve-6dc49468f8-rlzhx" podUID="8590a783-eb24-4759-ad74-1369f36aa9d2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded"
Oct 30 22:15:13.557426 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:13.532957    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9ndd9" podUID="81bd6415-f42f-446b-a64e-956f794b3956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": context deadline exceeded"
Oct 30 22:15:13.696426 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:13.581587    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-65e59f/simulator-w4-65e59f-kserve-85698bcd47-przch"
Oct 30 22:15:14.211420 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:14.204840    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-65e59f/premium-w4-65e59f-kserve-6dc49468f8-rlzhx"
Oct 30 22:15:14.211420 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:14.204928    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9ndd9"
Oct 30 22:15:14.211420 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:13.636787    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb0b93/simulator-w6-fb0b93-kserve-55b784b479-588bn" podUID="951a6f7e-a22e-40f2-ace2-1ce5d3bc9b49" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:15:14.225932 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:13.637025    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-65e59f/distinct-2-w4-65e59f-kserve-689566757d-bf98q" podUID="64e66fdb-056e-4086-a85a-8c02613584a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:15:14.234528 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:13.641979    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a00d9b/distinct-2-w1-a00d9b-kserve-68fd95957c-dmgnb" podUID="23a81fda-f7e3-4f91-8d96-9f19c7de6e09" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:15:14.243475 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:13.642153    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7bb1c2/simulator-w3-7bb1c2-kserve-675645869c-vcwqv" podUID="1fce9daf-68c9-41d0-825b-2012bb93c8ba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Oct 30 22:15:14.292428 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:14.281578    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-fb0b93/simulator-w6-fb0b93-kserve-55b784b479-588bn"
Oct 30 22:15:14.292428 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:14.281690    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-65e59f/distinct-2-w4-65e59f-kserve-689566757d-bf98q"
Oct 30 22:15:14.292428 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:14.281758    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a00d9b/distinct-2-w1-a00d9b-kserve-68fd95957c-dmgnb"
Oct 30 22:15:14.292428 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:14.281826    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-7bb1c2/simulator-w3-7bb1c2-kserve-675645869c-vcwqv"
Oct 30 22:15:14.292428 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:13.642381    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779f32/premium-w0-779f32-kserve-67978797c8-jpkvl" podUID="40138fb9-9a75-4458-a351-11eb61c4a1e3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Oct 30 22:15:14.409489 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:13.642654    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7bb1c2/distinct-w3-7bb1c2-kserve-8d695596c-8zgtw" podUID="36f0a6a2-54be-49bf-9202-502960cbb134" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:15:14.411079 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:13.839678    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7bb1c2/premium-w3-7bb1c2-kserve-7dbf875d88-dqxn5" podUID="a58c911f-e2eb-4820-a6b6-24afe78b5fd4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:15:14.412568 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:13.845533    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb0b93/distinct-2-w6-fb0b93-kserve-7d65bb4698-m6tkh" podUID="6fd276ef-522b-4bb4-b101-0a9c3fe9ecd3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:15:14.460483 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:13.948691    5678 patch_prober.go:29] interesting pod/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-q828p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:15:14.464187 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:14.464098    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-q828p" podUID="4e8c9108-98ce-4a09-a8a9-f13def9228df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:15:14.464984 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:14.461784    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-779f32/premium-w0-779f32-kserve-67978797c8-jpkvl"
Oct 30 22:15:14.465415 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:14.465376    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-7bb1c2/distinct-w3-7bb1c2-kserve-8d695596c-8zgtw"
Oct 30 22:15:14.465623 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:14.465608    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-7bb1c2/premium-w3-7bb1c2-kserve-7dbf875d88-dqxn5"
Oct 30 22:15:14.465818 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:14.465805    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-fb0b93/distinct-2-w6-fb0b93-kserve-7d65bb4698-m6tkh"
Oct 30 22:15:14.486235 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:14.486193    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-65e59f/premium-w4-65e59f-kserve-6dc49468f8-rlzhx" podUID="8590a783-eb24-4759-ad74-1369f36aa9d2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:15:14.486997 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:14.486873    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-65e59f/simulator-w4-65e59f-kserve-85698bcd47-przch" podUID="ee5e843b-17be-4623-8c95-dc07c8b7f506" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Oct 30 22:15:14.710425 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:14.707195    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9ndd9" podUID="81bd6415-f42f-446b-a64e-956f794b3956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": context deadline exceeded"
Oct 30 22:15:15.029521 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:15.029491    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-7bb1c2/distinct-w3-7bb1c2-kserve-8d695596c-8zgtw"
Oct 30 22:15:15.029775 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:15.029754    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9ndd9"
Oct 30 22:15:15.030132 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:15.030110    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-65e59f/simulator-w4-65e59f-kserve-85698bcd47-przch"
Oct 30 22:15:15.030896 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:15.030874    5678 scope.go:122] "RemoveContainer" containerID="3e4e2745a388848fc93877f346ac62fda7b332ae14e092271244fe26fe4df5bd"
Oct 30 22:15:15.151688 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:15.150457    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-7bb1c2/premium-w3-7bb1c2-kserve-7dbf875d88-dqxn5"
Oct 30 22:15:15.216456 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:15.216373    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-65e59f/distinct-2-w4-65e59f-kserve-689566757d-bf98q"
Oct 30 22:15:15.217589 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:15.216907    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb0b93/simulator-w6-fb0b93-kserve-55b784b479-588bn" podUID="951a6f7e-a22e-40f2-ace2-1ce5d3bc9b49" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:15:15.236052 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:15.236007    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a00d9b/distinct-2-w1-a00d9b-kserve-68fd95957c-dmgnb" podUID="23a81fda-f7e3-4f91-8d96-9f19c7de6e09" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:15:15.246026 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:15.245469    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7bb1c2/simulator-w3-7bb1c2-kserve-675645869c-vcwqv" podUID="1fce9daf-68c9-41d0-825b-2012bb93c8ba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:15:15.266563 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:15.266540    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-779f32/premium-w0-779f32-kserve-67978797c8-jpkvl"
Oct 30 22:15:15.286106 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:15.278556    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-fb0b93/distinct-2-w6-fb0b93-kserve-7d65bb4698-m6tkh"
Oct 30 22:15:15.287677 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:15.287656    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-65e59f/premium-w4-65e59f-kserve-6dc49468f8-rlzhx"
Oct 30 22:15:15.320417 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:15.319782    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-fb0b93/simulator-w6-fb0b93-kserve-55b784b479-588bn"
Oct 30 22:15:15.325417 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:15.324180    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-7bb1c2/simulator-w3-7bb1c2-kserve-675645869c-vcwqv"
Oct 30 22:15:15.350460 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:15.349878    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a00d9b/distinct-2-w1-a00d9b-kserve-68fd95957c-dmgnb"
Oct 30 22:15:15.515563 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:15.515537    5678 scope.go:122] "RemoveContainer" containerID="fb76ad9d82fb54d6c410614bfd81693b82c356f940a74cff79416971422949eb"
Oct 30 22:15:16.028204 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:16.010453    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-779f32-29889975-jb2qb" event={"ID":"3fc7905d-2cf4-4e42-a606-29ddd24529e1","Type":"ContainerStarted","Data":"2c4098a370b7c7a877754fee6fc6778863be97dc66819d76142dd7261d6f7b07"}
Oct 30 22:15:16.028204 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:16.015690    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9dbe6-29889975-x9bnd" event={"ID":"6a104de2-358d-47b3-b066-b8799c301c5c","Type":"ContainerStarted","Data":"000798bab19dffb7080039df42ba102b20c3a737fd402ddbfc27f5f7459e6663"}
Oct 30 22:15:16.537282 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:16.537243    5678 scope.go:122] "RemoveContainer" containerID="df7b006bc2f4cf441807a71b98030b016f6ad35ead7c4f3cf6c91f461c495308"
Oct 30 22:15:17.025939 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:17.025904    5678 generic.go:356] "Generic (PLEG): container finished" podID="3fc7905d-2cf4-4e42-a606-29ddd24529e1" containerID="2c4098a370b7c7a877754fee6fc6778863be97dc66819d76142dd7261d6f7b07" exitCode=60
Oct 30 22:15:17.026120 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:17.025976    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-779f32-29889975-jb2qb" event={"ID":"3fc7905d-2cf4-4e42-a606-29ddd24529e1","Type":"ContainerDied","Data":"2c4098a370b7c7a877754fee6fc6778863be97dc66819d76142dd7261d6f7b07"}
Oct 30 22:15:17.026120 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:17.026030    5678 scope.go:122] "RemoveContainer" containerID="3e4e2745a388848fc93877f346ac62fda7b332ae14e092271244fe26fe4df5bd"
Oct 30 22:15:17.026554 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:17.026536    5678 scope.go:122] "RemoveContainer" containerID="2c4098a370b7c7a877754fee6fc6778863be97dc66819d76142dd7261d6f7b07"
Oct 30 22:15:17.026986 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:15:17.026949    5678 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-779f32-29889975-jb2qb_odh-ai-gateway-infra(3fc7905d-2cf4-4e42-a606-29ddd24529e1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-779f32-29889975-jb2qb" podUID="3fc7905d-2cf4-4e42-a606-29ddd24529e1"
Oct 30 22:15:17.028264 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:17.028240    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb0b93-29889975-clfmz" event={"ID":"6c4cd55b-361c-49a8-9ea0-2bb45f97c28d","Type":"ContainerStarted","Data":"025a9b1f9ea1adee50ec0903bd390df34b6240f6239b24fcdc045d1fb5f2ba7d"}
Oct 30 22:15:17.030328 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:17.030303    5678 generic.go:356] "Generic (PLEG): container finished" podID="6a104de2-358d-47b3-b066-b8799c301c5c" containerID="000798bab19dffb7080039df42ba102b20c3a737fd402ddbfc27f5f7459e6663" exitCode=60
Oct 30 22:15:17.030459 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:17.030335    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9dbe6-29889975-x9bnd" event={"ID":"6a104de2-358d-47b3-b066-b8799c301c5c","Type":"ContainerDied","Data":"000798bab19dffb7080039df42ba102b20c3a737fd402ddbfc27f5f7459e6663"}
Oct 30 22:15:17.030738 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:17.030720    5678 scope.go:122] "RemoveContainer" containerID="000798bab19dffb7080039df42ba102b20c3a737fd402ddbfc27f5f7459e6663"
Oct 30 22:15:17.031013 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:15:17.030993    5678 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-c9dbe6-29889975-x9bnd_odh-ai-gateway-infra(6a104de2-358d-47b3-b066-b8799c301c5c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9dbe6-29889975-x9bnd" podUID="6a104de2-358d-47b3-b066-b8799c301c5c"
Oct 30 22:15:17.039687 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:17.039665    5678 scope.go:122] "RemoveContainer" containerID="fb76ad9d82fb54d6c410614bfd81693b82c356f940a74cff79416971422949eb"
Oct 30 22:15:17.518933 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:17.518902    5678 scope.go:122] "RemoveContainer" containerID="04af45220d43d287322cf4dc1807676eb59a4db8296b68660c5957e417f8a045"
Oct 30 22:15:18.040473 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:18.040443    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a00d9b-29889975-7nlt8" event={"ID":"569c8343-cc6a-437e-b488-0eabfac00d04","Type":"ContainerStarted","Data":"c037d4a0e20fa7a5ab89c672b1ef33224c41ae3dc9ebc53d80bf6cda55554c79"}
Oct 30 22:15:18.044041 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:18.044007    5678 generic.go:356] "Generic (PLEG): container finished" podID="6c4cd55b-361c-49a8-9ea0-2bb45f97c28d" containerID="025a9b1f9ea1adee50ec0903bd390df34b6240f6239b24fcdc045d1fb5f2ba7d" exitCode=60
Oct 30 22:15:18.044183 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:18.044075    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb0b93-29889975-clfmz" event={"ID":"6c4cd55b-361c-49a8-9ea0-2bb45f97c28d","Type":"ContainerDied","Data":"025a9b1f9ea1adee50ec0903bd390df34b6240f6239b24fcdc045d1fb5f2ba7d"}
Oct 30 22:15:18.044183 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:18.044123    5678 scope.go:122] "RemoveContainer" containerID="df7b006bc2f4cf441807a71b98030b016f6ad35ead7c4f3cf6c91f461c495308"
Oct 30 22:15:18.044506 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:18.044491    5678 scope.go:122] "RemoveContainer" containerID="025a9b1f9ea1adee50ec0903bd390df34b6240f6239b24fcdc045d1fb5f2ba7d"
Oct 30 22:15:18.044871 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:15:18.044850    5678 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-fb0b93-29889975-clfmz_odh-ai-gateway-infra(6c4cd55b-361c-49a8-9ea0-2bb45f97c28d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb0b93-29889975-clfmz" podUID="6c4cd55b-361c-49a8-9ea0-2bb45f97c28d"
Oct 30 22:15:18.359199 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:18.359164    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9dbe6-29889975-x9bnd"]
Oct 30 22:15:18.488025 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:18.482650    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb0b93-29889975-clfmz"]
Oct 30 22:15:18.576512 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:18.576480    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9dbe6-29889975-x9bnd"
Oct 30 22:15:18.609448 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:18.609347    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-779f32-29889975-jb2qb"]
Oct 30 22:15:18.632592 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:18.630714    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a104de2-358d-47b3-b066-b8799c301c5c-kube-api-access-qkcfj\" (UniqueName: \"kubernetes.io/projected/6a104de2-358d-47b3-b066-b8799c301c5c-kube-api-access-qkcfj\") pod \"6a104de2-358d-47b3-b066-b8799c301c5c\" (UID: \"6a104de2-358d-47b3-b066-b8799c301c5c\") "
Oct 30 22:15:18.632592 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:18.630834    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a104de2-358d-47b3-b066-b8799c301c5c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a104de2-358d-47b3-b066-b8799c301c5c-maas-api-service-ca\") pod \"6a104de2-358d-47b3-b066-b8799c301c5c\" (UID: \"6a104de2-358d-47b3-b066-b8799c301c5c\") "
Oct 30 22:15:18.632592 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:18.632312    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a104de2-358d-47b3-b066-b8799c301c5c-maas-api-service-ca" pod "6a104de2-358d-47b3-b066-b8799c301c5c" (UID: "6a104de2-358d-47b3-b066-b8799c301c5c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:15:18.634790 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:18.634760    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a104de2-358d-47b3-b066-b8799c301c5c-kube-api-access-qkcfj" pod "6a104de2-358d-47b3-b066-b8799c301c5c" (UID: "6a104de2-358d-47b3-b066-b8799c301c5c"). InnerVolumeSpecName "kube-api-access-qkcfj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:18.745766 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:18.745734    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkcfj\" (UniqueName: \"kubernetes.io/projected/6a104de2-358d-47b3-b066-b8799c301c5c-kube-api-access-qkcfj\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:18.745766 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:18.745764    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a104de2-358d-47b3-b066-b8799c301c5c-maas-api-service-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:18.859415 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:18.859372    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-779f32-29889975-jb2qb"
Oct 30 22:15:18.947949 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:18.947917    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fc7905d-2cf4-4e42-a606-29ddd24529e1-kube-api-access-8zsvw\" (UniqueName: \"kubernetes.io/projected/3fc7905d-2cf4-4e42-a606-29ddd24529e1-kube-api-access-8zsvw\") pod \"3fc7905d-2cf4-4e42-a606-29ddd24529e1\" (UID: \"3fc7905d-2cf4-4e42-a606-29ddd24529e1\") "
Oct 30 22:15:18.948149 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:18.947985    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3fc7905d-2cf4-4e42-a606-29ddd24529e1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fc7905d-2cf4-4e42-a606-29ddd24529e1-maas-api-service-ca\") pod \"3fc7905d-2cf4-4e42-a606-29ddd24529e1\" (UID: \"3fc7905d-2cf4-4e42-a606-29ddd24529e1\") "
Oct 30 22:15:18.948353 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:18.948329    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fc7905d-2cf4-4e42-a606-29ddd24529e1-maas-api-service-ca" pod "3fc7905d-2cf4-4e42-a606-29ddd24529e1" (UID: "3fc7905d-2cf4-4e42-a606-29ddd24529e1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:15:18.948570 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:18.948557    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fc7905d-2cf4-4e42-a606-29ddd24529e1-maas-api-service-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:18.950071 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:18.950052    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fc7905d-2cf4-4e42-a606-29ddd24529e1-kube-api-access-8zsvw" pod "3fc7905d-2cf4-4e42-a606-29ddd24529e1" (UID: "3fc7905d-2cf4-4e42-a606-29ddd24529e1"). InnerVolumeSpecName "kube-api-access-8zsvw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:19.052853 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:19.051674    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zsvw\" (UniqueName: \"kubernetes.io/projected/3fc7905d-2cf4-4e42-a606-29ddd24529e1-kube-api-access-8zsvw\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:19.070548 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:19.063186    5678 generic.go:356] "Generic (PLEG): container finished" podID="569c8343-cc6a-437e-b488-0eabfac00d04" containerID="c037d4a0e20fa7a5ab89c672b1ef33224c41ae3dc9ebc53d80bf6cda55554c79" exitCode=60
Oct 30 22:15:19.070548 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:19.063515    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a00d9b-29889975-7nlt8" event={"ID":"569c8343-cc6a-437e-b488-0eabfac00d04","Type":"ContainerDied","Data":"c037d4a0e20fa7a5ab89c672b1ef33224c41ae3dc9ebc53d80bf6cda55554c79"}
Oct 30 22:15:19.070548 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:19.063575    5678 scope.go:122] "RemoveContainer" containerID="04af45220d43d287322cf4dc1807676eb59a4db8296b68660c5957e417f8a045"
Oct 30 22:15:19.070548 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:19.064069    5678 scope.go:122] "RemoveContainer" containerID="c037d4a0e20fa7a5ab89c672b1ef33224c41ae3dc9ebc53d80bf6cda55554c79"
Oct 30 22:15:19.070548 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:15:19.067672    5678 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-a00d9b-29889975-7nlt8_odh-ai-gateway-infra(569c8343-cc6a-437e-b488-0eabfac00d04)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a00d9b-29889975-7nlt8" podUID="569c8343-cc6a-437e-b488-0eabfac00d04"
Oct 30 22:15:19.071615 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:19.071590    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-779f32-29889975-jb2qb"
Oct 30 22:15:19.072147 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:19.071593    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-779f32-29889975-jb2qb" event={"ID":"3fc7905d-2cf4-4e42-a606-29ddd24529e1","Type":"ContainerDied","Data":"66741602c593f117804e1713b6a4f34fccdf2be759a8c23cb4bc1de9a3252c29"}
Oct 30 22:15:19.083925 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:19.076434    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9dbe6-29889975-x9bnd"
Oct 30 22:15:19.083925 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:19.076457    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9dbe6-29889975-x9bnd" event={"ID":"6a104de2-358d-47b3-b066-b8799c301c5c","Type":"ContainerDied","Data":"6a4a44823e218a80069a91d11c54e03a42327cc0f8432c429d238ba37a5bbcbf"}
Oct 30 22:15:19.094875 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:19.094849    5678 scope.go:122] "RemoveContainer" containerID="2c4098a370b7c7a877754fee6fc6778863be97dc66819d76142dd7261d6f7b07"
Oct 30 22:15:19.120261 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:19.120231    5678 scope.go:122] "RemoveContainer" containerID="000798bab19dffb7080039df42ba102b20c3a737fd402ddbfc27f5f7459e6663"
Oct 30 22:15:19.177817 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:19.177783    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a00d9b-29889975-7nlt8"]
Oct 30 22:15:19.357161 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:19.357133    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb0b93-29889975-clfmz"
Oct 30 22:15:19.359056 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:19.359037    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c4cd55b-361c-49a8-9ea0-2bb45f97c28d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c4cd55b-361c-49a8-9ea0-2bb45f97c28d-maas-api-service-ca\") pod \"6c4cd55b-361c-49a8-9ea0-2bb45f97c28d\" (UID: \"6c4cd55b-361c-49a8-9ea0-2bb45f97c28d\") "
Oct 30 22:15:19.359267 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:19.359250    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c4cd55b-361c-49a8-9ea0-2bb45f97c28d-kube-api-access-gklz7\" (UniqueName: \"kubernetes.io/projected/6c4cd55b-361c-49a8-9ea0-2bb45f97c28d-kube-api-access-gklz7\") pod \"6c4cd55b-361c-49a8-9ea0-2bb45f97c28d\" (UID: \"6c4cd55b-361c-49a8-9ea0-2bb45f97c28d\") "
Oct 30 22:15:19.359529 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:19.359442    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c4cd55b-361c-49a8-9ea0-2bb45f97c28d-maas-api-service-ca" pod "6c4cd55b-361c-49a8-9ea0-2bb45f97c28d" (UID: "6c4cd55b-361c-49a8-9ea0-2bb45f97c28d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:15:19.359974 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:19.359954    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c4cd55b-361c-49a8-9ea0-2bb45f97c28d-maas-api-service-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:19.361512 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:19.361489    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c4cd55b-361c-49a8-9ea0-2bb45f97c28d-kube-api-access-gklz7" pod "6c4cd55b-361c-49a8-9ea0-2bb45f97c28d" (UID: "6c4cd55b-361c-49a8-9ea0-2bb45f97c28d"). InnerVolumeSpecName "kube-api-access-gklz7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:19.433898 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:19.433864    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-779f32-29889975-jb2qb"]
Oct 30 22:15:19.440493 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:19.440467    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-779f32-29889975-jb2qb"]
Oct 30 22:15:19.460928 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:19.460905    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gklz7\" (UniqueName: \"kubernetes.io/projected/6c4cd55b-361c-49a8-9ea0-2bb45f97c28d-kube-api-access-gklz7\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:19.551340 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:19.551255    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9dbe6-29889975-x9bnd"]
Oct 30 22:15:19.561467 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:19.560194    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9dbe6-29889975-x9bnd"]
Oct 30 22:15:20.087103 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:20.087019    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb0b93-29889975-clfmz" event={"ID":"6c4cd55b-361c-49a8-9ea0-2bb45f97c28d","Type":"ContainerDied","Data":"427485d83151453a0b891c25e048c2c0d051da9e68e4004f582233ad6ab89a74"}
Oct 30 22:15:20.087103 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:20.087044    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb0b93-29889975-clfmz"
Oct 30 22:15:20.087615 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:20.087111    5678 scope.go:122] "RemoveContainer" containerID="025a9b1f9ea1adee50ec0903bd390df34b6240f6239b24fcdc045d1fb5f2ba7d"
Oct 30 22:15:20.303645 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:20.303610    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a00d9b-29889975-7nlt8"
Oct 30 22:15:20.372726 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:20.372693    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb0b93-29889975-clfmz"]
Oct 30 22:15:20.378161 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:20.378134    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb0b93-29889975-clfmz"]
Oct 30 22:15:20.479378 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:20.479336    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/569c8343-cc6a-437e-b488-0eabfac00d04-kube-api-access-zz5wr\" (UniqueName: \"kubernetes.io/projected/569c8343-cc6a-437e-b488-0eabfac00d04-kube-api-access-zz5wr\") pod \"569c8343-cc6a-437e-b488-0eabfac00d04\" (UID: \"569c8343-cc6a-437e-b488-0eabfac00d04\") "
Oct 30 22:15:20.479623 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:20.479558    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/569c8343-cc6a-437e-b488-0eabfac00d04-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/569c8343-cc6a-437e-b488-0eabfac00d04-maas-api-service-ca\") pod \"569c8343-cc6a-437e-b488-0eabfac00d04\" (UID: \"569c8343-cc6a-437e-b488-0eabfac00d04\") "
Oct 30 22:15:20.487261 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:20.481283    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/569c8343-cc6a-437e-b488-0eabfac00d04-maas-api-service-ca" pod "569c8343-cc6a-437e-b488-0eabfac00d04" (UID: "569c8343-cc6a-437e-b488-0eabfac00d04"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:15:20.503796 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:20.503735    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/569c8343-cc6a-437e-b488-0eabfac00d04-kube-api-access-zz5wr" pod "569c8343-cc6a-437e-b488-0eabfac00d04" (UID: "569c8343-cc6a-437e-b488-0eabfac00d04"). InnerVolumeSpecName "kube-api-access-zz5wr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:20.530880 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:20.530223    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fc7905d-2cf4-4e42-a606-29ddd24529e1" path="/var/lib/kubelet/pods/3fc7905d-2cf4-4e42-a606-29ddd24529e1/volumes"
Oct 30 22:15:20.531120 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:20.531098    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a104de2-358d-47b3-b066-b8799c301c5c" path="/var/lib/kubelet/pods/6a104de2-358d-47b3-b066-b8799c301c5c/volumes"
Oct 30 22:15:20.532517 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:20.531637    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c4cd55b-361c-49a8-9ea0-2bb45f97c28d" path="/var/lib/kubelet/pods/6c4cd55b-361c-49a8-9ea0-2bb45f97c28d/volumes"
Oct 30 22:15:20.582724 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:20.582686    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zz5wr\" (UniqueName: \"kubernetes.io/projected/569c8343-cc6a-437e-b488-0eabfac00d04-kube-api-access-zz5wr\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:20.582724 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:20.582726    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/569c8343-cc6a-437e-b488-0eabfac00d04-maas-api-service-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:21.101591 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:21.101563    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a00d9b-29889975-7nlt8"
Oct 30 22:15:21.101591 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:21.101575    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a00d9b-29889975-7nlt8" event={"ID":"569c8343-cc6a-437e-b488-0eabfac00d04","Type":"ContainerDied","Data":"d9851305ac5891e51fc45d030f7208a500e2a8d770333ee7dfc404785fcd19e5"}
Oct 30 22:15:21.102044 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:21.101630    5678 scope.go:122] "RemoveContainer" containerID="c037d4a0e20fa7a5ab89c672b1ef33224c41ae3dc9ebc53d80bf6cda55554c79"
Oct 30 22:15:21.286427 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:21.285823    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a00d9b-29889975-7nlt8"]
Oct 30 22:15:21.326050 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:21.325202    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a00d9b-29889975-7nlt8"]
Oct 30 22:15:22.527093 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:22.527062    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="569c8343-cc6a-437e-b488-0eabfac00d04" path="/var/lib/kubelet/pods/569c8343-cc6a-437e-b488-0eabfac00d04/volumes"
Oct 30 22:15:22.668946 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:22.668914    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f42d803d-drk78"]
Oct 30 22:15:22.669772 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:22.669751    5678 cpu_manager.go:414] "containerMap: removing container" podUID="569c8343-cc6a-437e-b488-0eabfac00d04" containerName="cleanup"
Oct 30 22:15:22.669772 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:22.669770    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="569c8343-cc6a-437e-b488-0eabfac00d04" containerName="cleanup"
Oct 30 22:15:22.669923 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:22.669788    5678 cpu_manager.go:414] "containerMap: removing container" podUID="6c4cd55b-361c-49a8-9ea0-2bb45f97c28d" containerName="cleanup"
Oct 30 22:15:22.669923 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:22.669794    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c4cd55b-361c-49a8-9ea0-2bb45f97c28d" containerName="cleanup"
Oct 30 22:15:22.669923 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:22.669809    5678 cpu_manager.go:414] "containerMap: removing container" podUID="6a104de2-358d-47b3-b066-b8799c301c5c" containerName="cleanup"
Oct 30 22:15:22.669923 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:22.669814    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a104de2-358d-47b3-b066-b8799c301c5c" containerName="cleanup"
Oct 30 22:15:22.669923 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:22.669821    5678 cpu_manager.go:414] "containerMap: removing container" podUID="3fc7905d-2cf4-4e42-a606-29ddd24529e1" containerName="cleanup"
Oct 30 22:15:22.669923 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:22.669825    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fc7905d-2cf4-4e42-a606-29ddd24529e1" containerName="cleanup"
Oct 30 22:15:22.669923 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:22.669841    5678 cpu_manager.go:414] "containerMap: removing container" podUID="6a104de2-358d-47b3-b066-b8799c301c5c" containerName="cleanup"
Oct 30 22:15:22.669923 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:22.669845    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a104de2-358d-47b3-b066-b8799c301c5c" containerName="cleanup"
Oct 30 22:15:22.669923 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:22.669855    5678 cpu_manager.go:414] "containerMap: removing container" podUID="6c4cd55b-361c-49a8-9ea0-2bb45f97c28d" containerName="cleanup"
Oct 30 22:15:22.669923 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:22.669860    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c4cd55b-361c-49a8-9ea0-2bb45f97c28d" containerName="cleanup"
Oct 30 22:15:22.669923 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:22.669870    5678 cpu_manager.go:414] "containerMap: removing container" podUID="3fc7905d-2cf4-4e42-a606-29ddd24529e1" containerName="cleanup"
Oct 30 22:15:22.669923 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:22.669875    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fc7905d-2cf4-4e42-a606-29ddd24529e1" containerName="cleanup"
Oct 30 22:15:22.669923 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:22.669882    5678 cpu_manager.go:414] "containerMap: removing container" podUID="569c8343-cc6a-437e-b488-0eabfac00d04" containerName="cleanup"
Oct 30 22:15:22.669923 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:22.669886    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="569c8343-cc6a-437e-b488-0eabfac00d04" containerName="cleanup"
Oct 30 22:15:22.669923 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:22.669894    5678 cpu_manager.go:414] "containerMap: removing container" podUID="3fc7905d-2cf4-4e42-a606-29ddd24529e1" containerName="cleanup"
Oct 30 22:15:22.669923 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:22.669898    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fc7905d-2cf4-4e42-a606-29ddd24529e1" containerName="cleanup"
Oct 30 22:15:22.670368 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:22.669990    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="569c8343-cc6a-437e-b488-0eabfac00d04" containerName="cleanup"
Oct 30 22:15:22.670368 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:22.670000    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c4cd55b-361c-49a8-9ea0-2bb45f97c28d" containerName="cleanup"
Oct 30 22:15:22.670368 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:22.670007    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a104de2-358d-47b3-b066-b8799c301c5c" containerName="cleanup"
Oct 30 22:15:22.670368 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:22.670012    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fc7905d-2cf4-4e42-a606-29ddd24529e1" containerName="cleanup"
Oct 30 22:15:22.670368 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:22.670017    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="569c8343-cc6a-437e-b488-0eabfac00d04" containerName="cleanup"
Oct 30 22:15:22.670368 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:22.670024    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a104de2-358d-47b3-b066-b8799c301c5c" containerName="cleanup"
Oct 30 22:15:22.670368 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:22.670030    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c4cd55b-361c-49a8-9ea0-2bb45f97c28d" containerName="cleanup"
Oct 30 22:15:22.670368 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:22.670036    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fc7905d-2cf4-4e42-a606-29ddd24529e1" containerName="cleanup"
Oct 30 22:15:22.670368 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:22.670041    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="569c8343-cc6a-437e-b488-0eabfac00d04" containerName="cleanup"
Oct 30 22:15:22.670368 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:22.670048    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c4cd55b-361c-49a8-9ea0-2bb45f97c28d" containerName="cleanup"
Oct 30 22:15:22.675682 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:22.675657    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f42d803d-drk78"
Oct 30 22:15:22.729625 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:22.729596    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f42d803d-drk78"]
Oct 30 22:15:22.745629 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:22.745604    5678 scope.go:122] "RemoveContainer" containerID="60c895bad8fcf4df291affcfe3b4dcc47654b702a915ebaf1889d9d6886c8503"
Oct 30 22:15:22.759279 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:22.759251    5678 scope.go:122] "RemoveContainer" containerID="4d585125f281551cc003d2da6ef58346e1549a8697fd5835fddd501bf404e870"
Oct 30 22:15:22.770232 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:22.770206    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:15:22.770537 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:22.770512    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5rvcj\""
Oct 30 22:15:22.770623 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:22.770516    5678 scope.go:122] "RemoveContainer" containerID="22c312f89e8803cbe93f399a895e0e852500657988c7d3f9f7a6de3c45987fd5"
Oct 30 22:15:22.782018 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:22.781993    5678 scope.go:122] "RemoveContainer" containerID="66940fb240dfab6c3e14322d89fd1052a48ee723a31c2e5d378c08373b370c89"
Oct 30 22:15:22.791234 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:22.791218    5678 scope.go:122] "RemoveContainer" containerID="8225b867fc1f02c4c04f99fdbf7bd003b338a1e0a6cea99ec4d83fe5fca3f10d"
Oct 30 22:15:22.804252 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:22.804234    5678 scope.go:122] "RemoveContainer" containerID="a4fe8669512485d9b64a520c0fd490748e35be08af5c992e466808be94e56777"
Oct 30 22:15:22.813537 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:22.813520    5678 scope.go:122] "RemoveContainer" containerID="cc6d9e00b5fadbe1f0e6b9a1feef29fc005705fcf552bf8fe3aa35d9429e9e6c"
Oct 30 22:15:22.822697 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:22.822670    5678 scope.go:122] "RemoveContainer" containerID="4f4c41fd74ea0acea98a40bd9e2a4a127a511bc6317aafac7cd1dd3381a7f493"
Oct 30 22:15:22.832303 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:22.832285    5678 scope.go:122] "RemoveContainer" containerID="85191fe5eb85b0cb770344671e29fa69f048487d18093e591872e1f6e2193200"
Oct 30 22:15:22.832578 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:22.832550    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kndm9\" (UniqueName: \"kubernetes.io/projected/05eb2ec8-ad05-45bb-98a6-c074474fab94-kube-api-access-kndm9\") pod \"maas-api-revoke-sub-f42d803d-drk78\" (UID: \"05eb2ec8-ad05-45bb-98a6-c074474fab94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f42d803d-drk78"
Oct 30 22:15:22.832651 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:22.832631    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05eb2ec8-ad05-45bb-98a6-c074474fab94-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f42d803d-drk78\" (UID: \"05eb2ec8-ad05-45bb-98a6-c074474fab94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f42d803d-drk78"
Oct 30 22:15:22.832927 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:22.832904    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05eb2ec8-ad05-45bb-98a6-c074474fab94-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f42d803d-drk78\" (UID: \"05eb2ec8-ad05-45bb-98a6-c074474fab94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f42d803d-drk78"
Oct 30 22:15:22.841039 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:22.841019    5678 scope.go:122] "RemoveContainer" containerID="3de1003a8b4a38f0b001bb41db04fc0fd22e58b894f1b657c8cc060c7385ea59"
Oct 30 22:15:22.850260 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:22.850237    5678 scope.go:122] "RemoveContainer" containerID="dcc3ab333b217e67f6cb49a6c30b25b1f9ba35abbc667182318f0ac1d27b43f7"
Oct 30 22:15:22.934345 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:22.934322    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kndm9\" (UniqueName: \"kubernetes.io/projected/05eb2ec8-ad05-45bb-98a6-c074474fab94-kube-api-access-kndm9\") pod \"maas-api-revoke-sub-f42d803d-drk78\" (UID: \"05eb2ec8-ad05-45bb-98a6-c074474fab94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f42d803d-drk78"
Oct 30 22:15:22.934449 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:22.934362    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05eb2ec8-ad05-45bb-98a6-c074474fab94-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f42d803d-drk78\" (UID: \"05eb2ec8-ad05-45bb-98a6-c074474fab94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f42d803d-drk78"
Oct 30 22:15:22.934650 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:22.934628    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05eb2ec8-ad05-45bb-98a6-c074474fab94-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f42d803d-drk78\" (UID: \"05eb2ec8-ad05-45bb-98a6-c074474fab94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f42d803d-drk78"
Oct 30 22:15:22.935496 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:22.935474    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05eb2ec8-ad05-45bb-98a6-c074474fab94-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f42d803d-drk78\" (UID: \"05eb2ec8-ad05-45bb-98a6-c074474fab94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f42d803d-drk78"
Oct 30 22:15:22.936947 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:22.936926    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05eb2ec8-ad05-45bb-98a6-c074474fab94-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f42d803d-drk78\" (UID: \"05eb2ec8-ad05-45bb-98a6-c074474fab94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f42d803d-drk78"
Oct 30 22:15:22.993654 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:22.993618    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kndm9\" (UniqueName: \"kubernetes.io/projected/05eb2ec8-ad05-45bb-98a6-c074474fab94-kube-api-access-kndm9\") pod \"maas-api-revoke-sub-f42d803d-drk78\" (UID: \"05eb2ec8-ad05-45bb-98a6-c074474fab94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f42d803d-drk78"
Oct 30 22:15:23.291661 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:23.291628    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f42d803d-drk78"
Oct 30 22:15:23.690324 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:23.690297    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f42d803d-drk78"]
Oct 30 22:15:23.690900 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:15:23.690877    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05eb2ec8_ad05_45bb_98a6_c074474fab94.slice/crio-9b61fb85e352d6713a93f1ab53756b64398035178830542fedf7a28766882c9e WatchSource:0}: Error finding container 9b61fb85e352d6713a93f1ab53756b64398035178830542fedf7a28766882c9e: Status 404 returned error can't find the container with id 9b61fb85e352d6713a93f1ab53756b64398035178830542fedf7a28766882c9e
Oct 30 22:15:24.145960 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:24.145912    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f42d803d-drk78" event={"ID":"05eb2ec8-ad05-45bb-98a6-c074474fab94","Type":"ContainerStarted","Data":"948a3dbd938682d6ca7abf6a2a948bb2106cfeb1b08989d5ec85a3dac7617876"}
Oct 30 22:15:24.146138 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:24.146008    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f42d803d-drk78" event={"ID":"05eb2ec8-ad05-45bb-98a6-c074474fab94","Type":"ContainerStarted","Data":"9b61fb85e352d6713a93f1ab53756b64398035178830542fedf7a28766882c9e"}
Oct 30 22:15:24.261999 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:24.261952    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f42d803d-drk78" podStartSLOduration=2.261938639 podStartE2EDuration="2.261938639s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:24.258099515 +0000 UTC m=+1501.862407092" watchObservedRunningTime="2026-10-30 22:15:24.261938639 +0000 UTC m=+1501.866246227"
Oct 30 22:15:25.154623 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:25.154590    5678 generic.go:356] "Generic (PLEG): container finished" podID="05eb2ec8-ad05-45bb-98a6-c074474fab94" containerID="948a3dbd938682d6ca7abf6a2a948bb2106cfeb1b08989d5ec85a3dac7617876" exitCode=0
Oct 30 22:15:25.154998 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:25.154668    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f42d803d-drk78" event={"ID":"05eb2ec8-ad05-45bb-98a6-c074474fab94","Type":"ContainerDied","Data":"948a3dbd938682d6ca7abf6a2a948bb2106cfeb1b08989d5ec85a3dac7617876"}
Oct 30 22:15:26.334238 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:26.334212    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f42d803d-drk78"
Oct 30 22:15:26.386043 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:26.386013    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05eb2ec8-ad05-45bb-98a6-c074474fab94-kube-api-access-kndm9\" (UniqueName: \"kubernetes.io/projected/05eb2ec8-ad05-45bb-98a6-c074474fab94-kube-api-access-kndm9\") pod \"05eb2ec8-ad05-45bb-98a6-c074474fab94\" (UID: \"05eb2ec8-ad05-45bb-98a6-c074474fab94\") "
Oct 30 22:15:26.386253 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:26.386233    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05eb2ec8-ad05-45bb-98a6-c074474fab94-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05eb2ec8-ad05-45bb-98a6-c074474fab94-maas-api-xks-ca\") pod \"05eb2ec8-ad05-45bb-98a6-c074474fab94\" (UID: \"05eb2ec8-ad05-45bb-98a6-c074474fab94\") "
Oct 30 22:15:26.386332 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:26.386271    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05eb2ec8-ad05-45bb-98a6-c074474fab94-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05eb2ec8-ad05-45bb-98a6-c074474fab94-maas-api-openshift-ca\") pod \"05eb2ec8-ad05-45bb-98a6-c074474fab94\" (UID: \"05eb2ec8-ad05-45bb-98a6-c074474fab94\") "
Oct 30 22:15:26.386680 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:26.386659    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05eb2ec8-ad05-45bb-98a6-c074474fab94-maas-api-openshift-ca" pod "05eb2ec8-ad05-45bb-98a6-c074474fab94" (UID: "05eb2ec8-ad05-45bb-98a6-c074474fab94"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:15:26.386952 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:26.386936    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05eb2ec8-ad05-45bb-98a6-c074474fab94-maas-api-openshift-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:26.388134 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:26.388095    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05eb2ec8-ad05-45bb-98a6-c074474fab94-kube-api-access-kndm9" pod "05eb2ec8-ad05-45bb-98a6-c074474fab94" (UID: "05eb2ec8-ad05-45bb-98a6-c074474fab94"). InnerVolumeSpecName "kube-api-access-kndm9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:26.388255 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:26.388234    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05eb2ec8-ad05-45bb-98a6-c074474fab94-maas-api-xks-ca" pod "05eb2ec8-ad05-45bb-98a6-c074474fab94" (UID: "05eb2ec8-ad05-45bb-98a6-c074474fab94"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:15:26.488600 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:26.488530    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05eb2ec8-ad05-45bb-98a6-c074474fab94-maas-api-xks-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:26.488600 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:26.488564    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kndm9\" (UniqueName: \"kubernetes.io/projected/05eb2ec8-ad05-45bb-98a6-c074474fab94-kube-api-access-kndm9\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:27.171087 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:27.171055    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f42d803d-drk78"
Oct 30 22:15:27.171257 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:27.171061    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f42d803d-drk78" event={"ID":"05eb2ec8-ad05-45bb-98a6-c074474fab94","Type":"ContainerDied","Data":"9b61fb85e352d6713a93f1ab53756b64398035178830542fedf7a28766882c9e"}
Oct 30 22:15:27.171257 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:27.171172    5678 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9b61fb85e352d6713a93f1ab53756b64398035178830542fedf7a28766882c9e"
Oct 30 22:15:32.816498 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:32.816461    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-vzfdw"]
Oct 30 22:15:32.817163 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:32.817144    5678 cpu_manager.go:414] "containerMap: removing container" podUID="6a104de2-358d-47b3-b066-b8799c301c5c" containerName="cleanup"
Oct 30 22:15:32.817206 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:32.817164    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a104de2-358d-47b3-b066-b8799c301c5c" containerName="cleanup"
Oct 30 22:15:32.817206 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:32.817189    5678 cpu_manager.go:414] "containerMap: removing container" podUID="6c4cd55b-361c-49a8-9ea0-2bb45f97c28d" containerName="cleanup"
Oct 30 22:15:32.817206 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:32.817195    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c4cd55b-361c-49a8-9ea0-2bb45f97c28d" containerName="cleanup"
Oct 30 22:15:32.817305 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:32.817222    5678 cpu_manager.go:414] "containerMap: removing container" podUID="569c8343-cc6a-437e-b488-0eabfac00d04" containerName="cleanup"
Oct 30 22:15:32.817305 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:32.817228    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="569c8343-cc6a-437e-b488-0eabfac00d04" containerName="cleanup"
Oct 30 22:15:32.817305 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:32.817239    5678 cpu_manager.go:414] "containerMap: removing container" podUID="05eb2ec8-ad05-45bb-98a6-c074474fab94" containerName="revoke-keys"
Oct 30 22:15:32.817305 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:32.817244    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05eb2ec8-ad05-45bb-98a6-c074474fab94" containerName="revoke-keys"
Oct 30 22:15:32.817451 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:32.817357    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="05eb2ec8-ad05-45bb-98a6-c074474fab94" containerName="revoke-keys"
Oct 30 22:15:32.817451 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:32.817369    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fc7905d-2cf4-4e42-a606-29ddd24529e1" containerName="cleanup"
Oct 30 22:15:32.817451 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:32.817377    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a104de2-358d-47b3-b066-b8799c301c5c" containerName="cleanup"
Oct 30 22:15:32.821370 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:32.821344    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-vzfdw"
Oct 30 22:15:32.876250 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:32.876211    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24750677-9d8f-46f4-9f0e-acc3d230fb8f-istio-token\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-vzfdw\" (UID: \"24750677-9d8f-46f4-9f0e-acc3d230fb8f\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-vzfdw"
Oct 30 22:15:32.876440 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:32.876255    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24750677-9d8f-46f4-9f0e-acc3d230fb8f-istiod-ca-cert\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-vzfdw\" (UID: \"24750677-9d8f-46f4-9f0e-acc3d230fb8f\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-vzfdw"
Oct 30 22:15:32.876524 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:32.876453    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24750677-9d8f-46f4-9f0e-acc3d230fb8f-istio-podinfo\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-vzfdw\" (UID: \"24750677-9d8f-46f4-9f0e-acc3d230fb8f\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-vzfdw"
Oct 30 22:15:32.876584 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:32.876545    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mvvm\" (UniqueName: \"kubernetes.io/projected/24750677-9d8f-46f4-9f0e-acc3d230fb8f-kube-api-access-5mvvm\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-vzfdw\" (UID: \"24750677-9d8f-46f4-9f0e-acc3d230fb8f\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-vzfdw"
Oct 30 22:15:32.876641 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:32.876580    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24750677-9d8f-46f4-9f0e-acc3d230fb8f-istio-envoy\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-vzfdw\" (UID: \"24750677-9d8f-46f4-9f0e-acc3d230fb8f\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-vzfdw"
Oct 30 22:15:32.876698 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:32.876643    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24750677-9d8f-46f4-9f0e-acc3d230fb8f-workload-socket\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-vzfdw\" (UID: \"24750677-9d8f-46f4-9f0e-acc3d230fb8f\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-vzfdw"
Oct 30 22:15:32.876761 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:32.876747    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24750677-9d8f-46f4-9f0e-acc3d230fb8f-workload-certs\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-vzfdw\" (UID: \"24750677-9d8f-46f4-9f0e-acc3d230fb8f\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-vzfdw"
Oct 30 22:15:32.876808 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:32.876790    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24750677-9d8f-46f4-9f0e-acc3d230fb8f-istio-data\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-vzfdw\" (UID: \"24750677-9d8f-46f4-9f0e-acc3d230fb8f\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-vzfdw"
Oct 30 22:15:32.876918 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:32.876893    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24750677-9d8f-46f4-9f0e-acc3d230fb8f-credential-socket\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-vzfdw\" (UID: \"24750677-9d8f-46f4-9f0e-acc3d230fb8f\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-vzfdw"
Oct 30 22:15:32.926396 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:32.926362    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-vzfdw"]
Oct 30 22:15:32.978364 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:32.978329    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24750677-9d8f-46f4-9f0e-acc3d230fb8f-istio-data\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-vzfdw\" (UID: \"24750677-9d8f-46f4-9f0e-acc3d230fb8f\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-vzfdw"
Oct 30 22:15:32.978536 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:32.978387    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24750677-9d8f-46f4-9f0e-acc3d230fb8f-credential-socket\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-vzfdw\" (UID: \"24750677-9d8f-46f4-9f0e-acc3d230fb8f\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-vzfdw"
Oct 30 22:15:32.978579 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:32.978529    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24750677-9d8f-46f4-9f0e-acc3d230fb8f-istio-token\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-vzfdw\" (UID: \"24750677-9d8f-46f4-9f0e-acc3d230fb8f\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-vzfdw"
Oct 30 22:15:32.978579 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:32.978556    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24750677-9d8f-46f4-9f0e-acc3d230fb8f-istiod-ca-cert\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-vzfdw\" (UID: \"24750677-9d8f-46f4-9f0e-acc3d230fb8f\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-vzfdw"
Oct 30 22:15:32.978658 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:32.978643    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24750677-9d8f-46f4-9f0e-acc3d230fb8f-istio-podinfo\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-vzfdw\" (UID: \"24750677-9d8f-46f4-9f0e-acc3d230fb8f\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-vzfdw"
Oct 30 22:15:32.978723 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:32.978707    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mvvm\" (UniqueName: \"kubernetes.io/projected/24750677-9d8f-46f4-9f0e-acc3d230fb8f-kube-api-access-5mvvm\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-vzfdw\" (UID: \"24750677-9d8f-46f4-9f0e-acc3d230fb8f\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-vzfdw"
Oct 30 22:15:32.978776 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:32.978738    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24750677-9d8f-46f4-9f0e-acc3d230fb8f-istio-envoy\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-vzfdw\" (UID: \"24750677-9d8f-46f4-9f0e-acc3d230fb8f\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-vzfdw"
Oct 30 22:15:32.978828 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:32.978807    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24750677-9d8f-46f4-9f0e-acc3d230fb8f-credential-socket\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-vzfdw\" (UID: \"24750677-9d8f-46f4-9f0e-acc3d230fb8f\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-vzfdw"
Oct 30 22:15:32.978883 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:32.978733    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24750677-9d8f-46f4-9f0e-acc3d230fb8f-istio-data\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-vzfdw\" (UID: \"24750677-9d8f-46f4-9f0e-acc3d230fb8f\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-vzfdw"
Oct 30 22:15:32.978883 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:32.978818    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24750677-9d8f-46f4-9f0e-acc3d230fb8f-workload-socket\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-vzfdw\" (UID: \"24750677-9d8f-46f4-9f0e-acc3d230fb8f\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-vzfdw"
Oct 30 22:15:32.978992 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:32.978949    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24750677-9d8f-46f4-9f0e-acc3d230fb8f-workload-certs\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-vzfdw\" (UID: \"24750677-9d8f-46f4-9f0e-acc3d230fb8f\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-vzfdw"
Oct 30 22:15:32.979177 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:32.979145    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24750677-9d8f-46f4-9f0e-acc3d230fb8f-workload-socket\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-vzfdw\" (UID: \"24750677-9d8f-46f4-9f0e-acc3d230fb8f\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-vzfdw"
Oct 30 22:15:32.979297 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:32.979239    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24750677-9d8f-46f4-9f0e-acc3d230fb8f-workload-certs\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-vzfdw\" (UID: \"24750677-9d8f-46f4-9f0e-acc3d230fb8f\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-vzfdw"
Oct 30 22:15:32.979297 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:32.979247    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24750677-9d8f-46f4-9f0e-acc3d230fb8f-istiod-ca-cert\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-vzfdw\" (UID: \"24750677-9d8f-46f4-9f0e-acc3d230fb8f\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-vzfdw"
Oct 30 22:15:32.980862 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:32.980834    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24750677-9d8f-46f4-9f0e-acc3d230fb8f-istio-envoy\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-vzfdw\" (UID: \"24750677-9d8f-46f4-9f0e-acc3d230fb8f\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-vzfdw"
Oct 30 22:15:32.981049 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:32.981029    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24750677-9d8f-46f4-9f0e-acc3d230fb8f-istio-podinfo\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-vzfdw\" (UID: \"24750677-9d8f-46f4-9f0e-acc3d230fb8f\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-vzfdw"
Oct 30 22:15:33.038856 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:33.038824    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24750677-9d8f-46f4-9f0e-acc3d230fb8f-istio-token\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-vzfdw\" (UID: \"24750677-9d8f-46f4-9f0e-acc3d230fb8f\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-vzfdw"
Oct 30 22:15:33.039003 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:33.038987    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mvvm\" (UniqueName: \"kubernetes.io/projected/24750677-9d8f-46f4-9f0e-acc3d230fb8f-kube-api-access-5mvvm\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-vzfdw\" (UID: \"24750677-9d8f-46f4-9f0e-acc3d230fb8f\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-vzfdw"
Oct 30 22:15:33.135604 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:33.135573    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-vzfdw"
Oct 30 22:15:33.430381 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:33.430304    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e0b3fa3-6bjxv"]
Oct 30 22:15:33.435751 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:33.435731    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e0b3fa3-6bjxv"
Oct 30 22:15:33.485899 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:33.485869    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:15:33.486095 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:33.485925    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5rvcj\""
Oct 30 22:15:33.486204 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:33.486181    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b8b45be-c9f8-4918-a934-ab66867cdb7b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e0b3fa3-6bjxv\" (UID: \"0b8b45be-c9f8-4918-a934-ab66867cdb7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e0b3fa3-6bjxv"
Oct 30 22:15:33.486269 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:33.486254    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jqdf\" (UniqueName: \"kubernetes.io/projected/0b8b45be-c9f8-4918-a934-ab66867cdb7b-kube-api-access-7jqdf\") pod \"maas-api-revoke-sub-6e0b3fa3-6bjxv\" (UID: \"0b8b45be-c9f8-4918-a934-ab66867cdb7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e0b3fa3-6bjxv"
Oct 30 22:15:33.486328 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:33.486281    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b8b45be-c9f8-4918-a934-ab66867cdb7b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e0b3fa3-6bjxv\" (UID: \"0b8b45be-c9f8-4918-a934-ab66867cdb7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e0b3fa3-6bjxv"
Oct 30 22:15:33.490513 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:33.490489    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e0b3fa3-6bjxv"]
Oct 30 22:15:33.587940 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:33.587912    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jqdf\" (UniqueName: \"kubernetes.io/projected/0b8b45be-c9f8-4918-a934-ab66867cdb7b-kube-api-access-7jqdf\") pod \"maas-api-revoke-sub-6e0b3fa3-6bjxv\" (UID: \"0b8b45be-c9f8-4918-a934-ab66867cdb7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e0b3fa3-6bjxv"
Oct 30 22:15:33.588190 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:33.587948    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b8b45be-c9f8-4918-a934-ab66867cdb7b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e0b3fa3-6bjxv\" (UID: \"0b8b45be-c9f8-4918-a934-ab66867cdb7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e0b3fa3-6bjxv"
Oct 30 22:15:33.588274 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:33.588225    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b8b45be-c9f8-4918-a934-ab66867cdb7b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e0b3fa3-6bjxv\" (UID: \"0b8b45be-c9f8-4918-a934-ab66867cdb7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e0b3fa3-6bjxv"
Oct 30 22:15:33.588858 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:33.588831    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b8b45be-c9f8-4918-a934-ab66867cdb7b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e0b3fa3-6bjxv\" (UID: \"0b8b45be-c9f8-4918-a934-ab66867cdb7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e0b3fa3-6bjxv"
Oct 30 22:15:33.590963 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:33.590946    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b8b45be-c9f8-4918-a934-ab66867cdb7b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e0b3fa3-6bjxv\" (UID: \"0b8b45be-c9f8-4918-a934-ab66867cdb7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e0b3fa3-6bjxv"
Oct 30 22:15:33.593243 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:15:33.593218    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24750677_9d8f_46f4_9f0e_acc3d230fb8f.slice/crio-62851d4513aef519293e952521ee38c3e61228efc0787c3c8bbcb31de52972a7 WatchSource:0}: Error finding container 62851d4513aef519293e952521ee38c3e61228efc0787c3c8bbcb31de52972a7: Status 404 returned error can't find the container with id 62851d4513aef519293e952521ee38c3e61228efc0787c3c8bbcb31de52972a7
Oct 30 22:15:33.594468 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:33.594446    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-vzfdw"]
Oct 30 22:15:33.595693 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:33.595667    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:15:33.595756 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:33.595743    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:15:33.595793 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:33.595780    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:15:33.646344 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:33.646319    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jqdf\" (UniqueName: \"kubernetes.io/projected/0b8b45be-c9f8-4918-a934-ab66867cdb7b-kube-api-access-7jqdf\") pod \"maas-api-revoke-sub-6e0b3fa3-6bjxv\" (UID: \"0b8b45be-c9f8-4918-a934-ab66867cdb7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e0b3fa3-6bjxv"
Oct 30 22:15:33.750416 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:33.750336    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e0b3fa3-6bjxv"
Oct 30 22:15:34.158723 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:15:34.158558    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b8b45be_c9f8_4918_a934_ab66867cdb7b.slice/crio-fd9189dd0ffe94bcf6af8e3439227f9e36287ee027fc4a3f6458cc45c216af28 WatchSource:0}: Error finding container fd9189dd0ffe94bcf6af8e3439227f9e36287ee027fc4a3f6458cc45c216af28: Status 404 returned error can't find the container with id fd9189dd0ffe94bcf6af8e3439227f9e36287ee027fc4a3f6458cc45c216af28
Oct 30 22:15:34.159046 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:34.158719    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e0b3fa3-6bjxv"]
Oct 30 22:15:34.259146 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:34.259106    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e0b3fa3-6bjxv" event={"ID":"0b8b45be-c9f8-4918-a934-ab66867cdb7b","Type":"ContainerStarted","Data":"fd9189dd0ffe94bcf6af8e3439227f9e36287ee027fc4a3f6458cc45c216af28"}
Oct 30 22:15:34.260899 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:34.260872    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-vzfdw" event={"ID":"24750677-9d8f-46f4-9f0e-acc3d230fb8f","Type":"ContainerStarted","Data":"e9bc67d5b4029824f09867d4654c397405e4d862427cc702d0ac4bb628896208"}
Oct 30 22:15:34.261014 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:34.260918    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-vzfdw" event={"ID":"24750677-9d8f-46f4-9f0e-acc3d230fb8f","Type":"ContainerStarted","Data":"62851d4513aef519293e952521ee38c3e61228efc0787c3c8bbcb31de52972a7"}
Oct 30 22:15:34.378513 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:34.378454    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-vzfdw" podStartSLOduration=2.378440014 podStartE2EDuration="2.378440014s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:34.376547998 +0000 UTC m=+1511.980855577" watchObservedRunningTime="2026-10-30 22:15:34.378440014 +0000 UTC m=+1511.982747566"
Oct 30 22:15:35.150427 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:35.149056    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-vzfdw"
Oct 30 22:15:35.150720 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:35.150693    5678 patch_prober.go:29] interesting pod/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-vzfdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body=
Oct 30 22:15:35.150844 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:35.150750    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-vzfdw" podUID="24750677-9d8f-46f4-9f0e-acc3d230fb8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused"
Oct 30 22:15:35.271518 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:35.271486    5678 generic.go:356] "Generic (PLEG): container finished" podID="0b8b45be-c9f8-4918-a934-ab66867cdb7b" containerID="89a9a36b09749a3312ec29b11c219826265df15a4d3a24bc9bda056f5cca3d38" exitCode=0
Oct 30 22:15:35.271874 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:35.271575    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e0b3fa3-6bjxv" event={"ID":"0b8b45be-c9f8-4918-a934-ab66867cdb7b","Type":"ContainerDied","Data":"89a9a36b09749a3312ec29b11c219826265df15a4d3a24bc9bda056f5cca3d38"}
Oct 30 22:15:36.142425 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:36.136768    5678 patch_prober.go:29] interesting pod/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-vzfdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body=
Oct 30 22:15:36.142425 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:36.136831    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-vzfdw" podUID="24750677-9d8f-46f4-9f0e-acc3d230fb8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused"
Oct 30 22:15:36.493802 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:36.493774    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e0b3fa3-6bjxv"
Oct 30 22:15:36.656130 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:36.656091    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b8b45be-c9f8-4918-a934-ab66867cdb7b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b8b45be-c9f8-4918-a934-ab66867cdb7b-maas-api-xks-ca\") pod \"0b8b45be-c9f8-4918-a934-ab66867cdb7b\" (UID: \"0b8b45be-c9f8-4918-a934-ab66867cdb7b\") "
Oct 30 22:15:36.657016 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:36.656992    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b8b45be-c9f8-4918-a934-ab66867cdb7b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b8b45be-c9f8-4918-a934-ab66867cdb7b-maas-api-openshift-ca\") pod \"0b8b45be-c9f8-4918-a934-ab66867cdb7b\" (UID: \"0b8b45be-c9f8-4918-a934-ab66867cdb7b\") "
Oct 30 22:15:36.657120 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:36.657026    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b8b45be-c9f8-4918-a934-ab66867cdb7b-kube-api-access-7jqdf\" (UniqueName: \"kubernetes.io/projected/0b8b45be-c9f8-4918-a934-ab66867cdb7b-kube-api-access-7jqdf\") pod \"0b8b45be-c9f8-4918-a934-ab66867cdb7b\" (UID: \"0b8b45be-c9f8-4918-a934-ab66867cdb7b\") "
Oct 30 22:15:36.657785 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:36.657756    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b8b45be-c9f8-4918-a934-ab66867cdb7b-maas-api-openshift-ca" pod "0b8b45be-c9f8-4918-a934-ab66867cdb7b" (UID: "0b8b45be-c9f8-4918-a934-ab66867cdb7b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:15:36.658503 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:36.658478    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b8b45be-c9f8-4918-a934-ab66867cdb7b-maas-api-xks-ca" pod "0b8b45be-c9f8-4918-a934-ab66867cdb7b" (UID: "0b8b45be-c9f8-4918-a934-ab66867cdb7b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:15:36.659798 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:36.659771    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b8b45be-c9f8-4918-a934-ab66867cdb7b-kube-api-access-7jqdf" pod "0b8b45be-c9f8-4918-a934-ab66867cdb7b" (UID: "0b8b45be-c9f8-4918-a934-ab66867cdb7b"). InnerVolumeSpecName "kube-api-access-7jqdf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:36.758491 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:36.758385    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b8b45be-c9f8-4918-a934-ab66867cdb7b-maas-api-openshift-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:36.758491 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:36.758441    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jqdf\" (UniqueName: \"kubernetes.io/projected/0b8b45be-c9f8-4918-a934-ab66867cdb7b-kube-api-access-7jqdf\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:36.758491 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:36.758453    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b8b45be-c9f8-4918-a934-ab66867cdb7b-maas-api-xks-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:37.144677 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:37.144636    5678 patch_prober.go:29] interesting pod/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-vzfdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body=
Oct 30 22:15:37.144886 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:37.144703    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-vzfdw" podUID="24750677-9d8f-46f4-9f0e-acc3d230fb8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused"
Oct 30 22:15:37.294667 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:37.291471    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e0b3fa3-6bjxv"
Oct 30 22:15:37.294667 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:37.291522    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e0b3fa3-6bjxv" event={"ID":"0b8b45be-c9f8-4918-a934-ab66867cdb7b","Type":"ContainerDied","Data":"fd9189dd0ffe94bcf6af8e3439227f9e36287ee027fc4a3f6458cc45c216af28"}
Oct 30 22:15:37.294667 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:37.291581    5678 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fd9189dd0ffe94bcf6af8e3439227f9e36287ee027fc4a3f6458cc45c216af28"
Oct 30 22:15:38.146579 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:38.146542    5678 patch_prober.go:29] interesting pod/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-vzfdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body=
Oct 30 22:15:38.147010 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:38.146619    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-vzfdw" podUID="24750677-9d8f-46f4-9f0e-acc3d230fb8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused"
Oct 30 22:15:38.856587 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:38.856554    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72eaded0-sqfvh"]
Oct 30 22:15:38.857207 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:38.857193    5678 cpu_manager.go:414] "containerMap: removing container" podUID="0b8b45be-c9f8-4918-a934-ab66867cdb7b" containerName="revoke-keys"
Oct 30 22:15:38.857255 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:38.857207    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b8b45be-c9f8-4918-a934-ab66867cdb7b" containerName="revoke-keys"
Oct 30 22:15:38.857324 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:38.857314    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b8b45be-c9f8-4918-a934-ab66867cdb7b" containerName="revoke-keys"
Oct 30 22:15:38.861539 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:38.861515    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72eaded0-sqfvh"
Oct 30 22:15:38.912183 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:38.912149    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72eaded0-sqfvh"]
Oct 30 22:15:38.953470 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:38.953376    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:15:38.953771 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:38.953714    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5rvcj\""
Oct 30 22:15:38.989754 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:38.989719    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f1f5fbe-f541-45aa-89b1-a0971f43c2cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-72eaded0-sqfvh\" (UID: \"1f1f5fbe-f541-45aa-89b1-a0971f43c2cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72eaded0-sqfvh"
Oct 30 22:15:38.989926 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:38.989766    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f1f5fbe-f541-45aa-89b1-a0971f43c2cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-72eaded0-sqfvh\" (UID: \"1f1f5fbe-f541-45aa-89b1-a0971f43c2cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72eaded0-sqfvh"
Oct 30 22:15:38.989926 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:38.989812    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84r5h\" (UniqueName: \"kubernetes.io/projected/1f1f5fbe-f541-45aa-89b1-a0971f43c2cf-kube-api-access-84r5h\") pod \"maas-api-revoke-sub-72eaded0-sqfvh\" (UID: \"1f1f5fbe-f541-45aa-89b1-a0971f43c2cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72eaded0-sqfvh"
Oct 30 22:15:39.091499 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:39.091460    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f1f5fbe-f541-45aa-89b1-a0971f43c2cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-72eaded0-sqfvh\" (UID: \"1f1f5fbe-f541-45aa-89b1-a0971f43c2cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72eaded0-sqfvh"
Oct 30 22:15:39.091700 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:39.091532    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84r5h\" (UniqueName: \"kubernetes.io/projected/1f1f5fbe-f541-45aa-89b1-a0971f43c2cf-kube-api-access-84r5h\") pod \"maas-api-revoke-sub-72eaded0-sqfvh\" (UID: \"1f1f5fbe-f541-45aa-89b1-a0971f43c2cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72eaded0-sqfvh"
Oct 30 22:15:39.091776 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:39.091763    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f1f5fbe-f541-45aa-89b1-a0971f43c2cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-72eaded0-sqfvh\" (UID: \"1f1f5fbe-f541-45aa-89b1-a0971f43c2cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72eaded0-sqfvh"
Oct 30 22:15:39.092366 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:39.092339    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f1f5fbe-f541-45aa-89b1-a0971f43c2cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-72eaded0-sqfvh\" (UID: \"1f1f5fbe-f541-45aa-89b1-a0971f43c2cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72eaded0-sqfvh"
Oct 30 22:15:39.093890 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:39.093869    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f1f5fbe-f541-45aa-89b1-a0971f43c2cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-72eaded0-sqfvh\" (UID: \"1f1f5fbe-f541-45aa-89b1-a0971f43c2cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72eaded0-sqfvh"
Oct 30 22:15:39.146943 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:39.146903    5678 patch_prober.go:29] interesting pod/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-vzfdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body=
Oct 30 22:15:39.147392 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:39.146983    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-vzfdw" podUID="24750677-9d8f-46f4-9f0e-acc3d230fb8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused"
Oct 30 22:15:39.152229 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:39.152206    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84r5h\" (UniqueName: \"kubernetes.io/projected/1f1f5fbe-f541-45aa-89b1-a0971f43c2cf-kube-api-access-84r5h\") pod \"maas-api-revoke-sub-72eaded0-sqfvh\" (UID: \"1f1f5fbe-f541-45aa-89b1-a0971f43c2cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72eaded0-sqfvh"
Oct 30 22:15:39.176931 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:39.176901    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72eaded0-sqfvh"
Oct 30 22:15:39.597647 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:15:39.597609    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f1f5fbe_f541_45aa_89b1_a0971f43c2cf.slice/crio-7fb5b807136918325ecfbd1749ebdea726987feb382549d23f15447534f25f91 WatchSource:0}: Error finding container 7fb5b807136918325ecfbd1749ebdea726987feb382549d23f15447534f25f91: Status 404 returned error can't find the container with id 7fb5b807136918325ecfbd1749ebdea726987feb382549d23f15447534f25f91
Oct 30 22:15:39.613694 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:39.613660    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72eaded0-sqfvh"]
Oct 30 22:15:40.135867 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:40.135836    5678 patch_prober.go:29] interesting pod/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-vzfdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body=
Oct 30 22:15:40.136049 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:40.135888    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-vzfdw" podUID="24750677-9d8f-46f4-9f0e-acc3d230fb8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused"
Oct 30 22:15:40.324008 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:40.323910    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72eaded0-sqfvh" event={"ID":"1f1f5fbe-f541-45aa-89b1-a0971f43c2cf","Type":"ContainerStarted","Data":"a3f9a432af4e2b0be712b48b2f0187fa8f12ddcff1719b9854d8cde91c50cf28"}
Oct 30 22:15:40.324008 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:40.323974    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72eaded0-sqfvh" event={"ID":"1f1f5fbe-f541-45aa-89b1-a0971f43c2cf","Type":"ContainerStarted","Data":"7fb5b807136918325ecfbd1749ebdea726987feb382549d23f15447534f25f91"}
Oct 30 22:15:40.439863 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:40.439809    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72eaded0-sqfvh" podStartSLOduration=2.439794055 podStartE2EDuration="2.439794055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:40.435379622 +0000 UTC m=+1518.039687186" watchObservedRunningTime="2026-10-30 22:15:40.439794055 +0000 UTC m=+1518.044101634"
Oct 30 22:15:40.574249 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:15:40.574155    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 22:15:41.140371 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:41.140339    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-vzfdw"
Oct 30 22:15:41.140845 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:41.140830    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-vzfdw"
Oct 30 22:15:41.141561 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:41.141540    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-vzfdw"
Oct 30 22:15:41.334110 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:41.334075    5678 generic.go:356] "Generic (PLEG): container finished" podID="1f1f5fbe-f541-45aa-89b1-a0971f43c2cf" containerID="a3f9a432af4e2b0be712b48b2f0187fa8f12ddcff1719b9854d8cde91c50cf28" exitCode=0
Oct 30 22:15:41.334617 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:41.334169    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72eaded0-sqfvh" event={"ID":"1f1f5fbe-f541-45aa-89b1-a0971f43c2cf","Type":"ContainerDied","Data":"a3f9a432af4e2b0be712b48b2f0187fa8f12ddcff1719b9854d8cde91c50cf28"}
Oct 30 22:15:42.506281 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:42.506259    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72eaded0-sqfvh"
Oct 30 22:15:42.559349 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:42.559319    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f1f5fbe-f541-45aa-89b1-a0971f43c2cf-kube-api-access-84r5h\" (UniqueName: \"kubernetes.io/projected/1f1f5fbe-f541-45aa-89b1-a0971f43c2cf-kube-api-access-84r5h\") pod \"1f1f5fbe-f541-45aa-89b1-a0971f43c2cf\" (UID: \"1f1f5fbe-f541-45aa-89b1-a0971f43c2cf\") "
Oct 30 22:15:42.559487 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:42.559358    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f1f5fbe-f541-45aa-89b1-a0971f43c2cf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f1f5fbe-f541-45aa-89b1-a0971f43c2cf-maas-api-openshift-ca\") pod \"1f1f5fbe-f541-45aa-89b1-a0971f43c2cf\" (UID: \"1f1f5fbe-f541-45aa-89b1-a0971f43c2cf\") "
Oct 30 22:15:42.559710 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:42.559687    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f1f5fbe-f541-45aa-89b1-a0971f43c2cf-maas-api-openshift-ca" pod "1f1f5fbe-f541-45aa-89b1-a0971f43c2cf" (UID: "1f1f5fbe-f541-45aa-89b1-a0971f43c2cf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:15:42.559752 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:42.559736    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f1f5fbe-f541-45aa-89b1-a0971f43c2cf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f1f5fbe-f541-45aa-89b1-a0971f43c2cf-maas-api-xks-ca\") pod \"1f1f5fbe-f541-45aa-89b1-a0971f43c2cf\" (UID: \"1f1f5fbe-f541-45aa-89b1-a0971f43c2cf\") "
Oct 30 22:15:42.560251 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:42.560229    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f1f5fbe-f541-45aa-89b1-a0971f43c2cf-maas-api-openshift-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:42.561649 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:42.561628    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f1f5fbe-f541-45aa-89b1-a0971f43c2cf-kube-api-access-84r5h" pod "1f1f5fbe-f541-45aa-89b1-a0971f43c2cf" (UID: "1f1f5fbe-f541-45aa-89b1-a0971f43c2cf"). InnerVolumeSpecName "kube-api-access-84r5h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:42.561913 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:42.561893    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f1f5fbe-f541-45aa-89b1-a0971f43c2cf-maas-api-xks-ca" pod "1f1f5fbe-f541-45aa-89b1-a0971f43c2cf" (UID: "1f1f5fbe-f541-45aa-89b1-a0971f43c2cf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:15:42.663383 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:42.663310    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f1f5fbe-f541-45aa-89b1-a0971f43c2cf-maas-api-xks-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:42.663642 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:42.663621    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84r5h\" (UniqueName: \"kubernetes.io/projected/1f1f5fbe-f541-45aa-89b1-a0971f43c2cf-kube-api-access-84r5h\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:43.351267 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:43.351242    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72eaded0-sqfvh"
Oct 30 22:15:43.351500 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:43.351226    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72eaded0-sqfvh" event={"ID":"1f1f5fbe-f541-45aa-89b1-a0971f43c2cf","Type":"ContainerDied","Data":"7fb5b807136918325ecfbd1749ebdea726987feb382549d23f15447534f25f91"}
Oct 30 22:15:43.351500 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:43.351441    5678 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7fb5b807136918325ecfbd1749ebdea726987feb382549d23f15447534f25f91"
Oct 30 22:15:48.438183 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:48.438148    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-67184f2a-59c88bbd6c-j769f"]
Oct 30 22:15:48.438925 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:48.438906    5678 cpu_manager.go:414] "containerMap: removing container" podUID="1f1f5fbe-f541-45aa-89b1-a0971f43c2cf" containerName="revoke-keys"
Oct 30 22:15:48.438983 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:48.438928    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f1f5fbe-f541-45aa-89b1-a0971f43c2cf" containerName="revoke-keys"
Oct 30 22:15:48.439058 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:48.439048    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f1f5fbe-f541-45aa-89b1-a0971f43c2cf" containerName="revoke-keys"
Oct 30 22:15:48.442888 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:48.442870    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-67184f2a-59c88bbd6c-j769f"
Oct 30 22:15:48.540309 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:48.540283    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-67184f2a-serving-cert\""
Oct 30 22:15:48.545071 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:48.545048    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-67184f2a-59c88bbd6c-j769f"]
Oct 30 22:15:48.545732 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:48.545712    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3129fc82-b397-4f74-a39c-f45e123c4ad1-maas-api-tls\") pod \"maas-api-e2e-migrate-67184f2a-59c88bbd6c-j769f\" (UID: \"3129fc82-b397-4f74-a39c-f45e123c4ad1\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-67184f2a-59c88bbd6c-j769f"
Oct 30 22:15:48.545894 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:48.545867    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3129fc82-b397-4f74-a39c-f45e123c4ad1-metrics-certs\") pod \"maas-api-e2e-migrate-67184f2a-59c88bbd6c-j769f\" (UID: \"3129fc82-b397-4f74-a39c-f45e123c4ad1\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-67184f2a-59c88bbd6c-j769f"
Oct 30 22:15:48.546011 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:48.545995    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3129fc82-b397-4f74-a39c-f45e123c4ad1-openshift-service-ca\") pod \"maas-api-e2e-migrate-67184f2a-59c88bbd6c-j769f\" (UID: \"3129fc82-b397-4f74-a39c-f45e123c4ad1\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-67184f2a-59c88bbd6c-j769f"
Oct 30 22:15:48.546084 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:48.546071    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsrjh\" (UniqueName: \"kubernetes.io/projected/3129fc82-b397-4f74-a39c-f45e123c4ad1-kube-api-access-fsrjh\") pod \"maas-api-e2e-migrate-67184f2a-59c88bbd6c-j769f\" (UID: \"3129fc82-b397-4f74-a39c-f45e123c4ad1\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-67184f2a-59c88bbd6c-j769f"
Oct 30 22:15:48.647462 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:48.647414    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3129fc82-b397-4f74-a39c-f45e123c4ad1-metrics-certs\") pod \"maas-api-e2e-migrate-67184f2a-59c88bbd6c-j769f\" (UID: \"3129fc82-b397-4f74-a39c-f45e123c4ad1\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-67184f2a-59c88bbd6c-j769f"
Oct 30 22:15:48.647635 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:48.647538    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3129fc82-b397-4f74-a39c-f45e123c4ad1-openshift-service-ca\") pod \"maas-api-e2e-migrate-67184f2a-59c88bbd6c-j769f\" (UID: \"3129fc82-b397-4f74-a39c-f45e123c4ad1\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-67184f2a-59c88bbd6c-j769f"
Oct 30 22:15:48.647635 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:48.647598    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsrjh\" (UniqueName: \"kubernetes.io/projected/3129fc82-b397-4f74-a39c-f45e123c4ad1-kube-api-access-fsrjh\") pod \"maas-api-e2e-migrate-67184f2a-59c88bbd6c-j769f\" (UID: \"3129fc82-b397-4f74-a39c-f45e123c4ad1\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-67184f2a-59c88bbd6c-j769f"
Oct 30 22:15:48.647763 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:48.647735    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3129fc82-b397-4f74-a39c-f45e123c4ad1-maas-api-tls\") pod \"maas-api-e2e-migrate-67184f2a-59c88bbd6c-j769f\" (UID: \"3129fc82-b397-4f74-a39c-f45e123c4ad1\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-67184f2a-59c88bbd6c-j769f"
Oct 30 22:15:48.647886 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:15:48.647820    5678 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-67184f2a-serving-cert: secret "maas-api-e2e-migrate-67184f2a-serving-cert" not found
Oct 30 22:15:48.647937 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:15:48.647917    5678 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3129fc82-b397-4f74-a39c-f45e123c4ad1-maas-api-tls podName:3129fc82-b397-4f74-a39c-f45e123c4ad1 nodeName:}" failed. No retries permitted until 2026-10-30 22:15:49.147893958 +0000 UTC m=+1526.752201509 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3129fc82-b397-4f74-a39c-f45e123c4ad1-maas-api-tls") pod "maas-api-e2e-migrate-67184f2a-59c88bbd6c-j769f" (UID: "3129fc82-b397-4f74-a39c-f45e123c4ad1") : secret "maas-api-e2e-migrate-67184f2a-serving-cert" not found
Oct 30 22:15:48.648304 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:48.648283    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3129fc82-b397-4f74-a39c-f45e123c4ad1-openshift-service-ca\") pod \"maas-api-e2e-migrate-67184f2a-59c88bbd6c-j769f\" (UID: \"3129fc82-b397-4f74-a39c-f45e123c4ad1\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-67184f2a-59c88bbd6c-j769f"
Oct 30 22:15:48.650147 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:48.650126    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3129fc82-b397-4f74-a39c-f45e123c4ad1-metrics-certs\") pod \"maas-api-e2e-migrate-67184f2a-59c88bbd6c-j769f\" (UID: \"3129fc82-b397-4f74-a39c-f45e123c4ad1\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-67184f2a-59c88bbd6c-j769f"
Oct 30 22:15:48.707144 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:48.707079    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsrjh\" (UniqueName: \"kubernetes.io/projected/3129fc82-b397-4f74-a39c-f45e123c4ad1-kube-api-access-fsrjh\") pod \"maas-api-e2e-migrate-67184f2a-59c88bbd6c-j769f\" (UID: \"3129fc82-b397-4f74-a39c-f45e123c4ad1\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-67184f2a-59c88bbd6c-j769f"
Oct 30 22:15:49.157032 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:49.157000    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3129fc82-b397-4f74-a39c-f45e123c4ad1-maas-api-tls\") pod \"maas-api-e2e-migrate-67184f2a-59c88bbd6c-j769f\" (UID: \"3129fc82-b397-4f74-a39c-f45e123c4ad1\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-67184f2a-59c88bbd6c-j769f"
Oct 30 22:15:49.159743 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:49.159683    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3129fc82-b397-4f74-a39c-f45e123c4ad1-maas-api-tls\") pod \"maas-api-e2e-migrate-67184f2a-59c88bbd6c-j769f\" (UID: \"3129fc82-b397-4f74-a39c-f45e123c4ad1\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-67184f2a-59c88bbd6c-j769f"
Oct 30 22:15:49.360428 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:49.359885    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-67184f2a-59c88bbd6c-j769f"
Oct 30 22:15:49.799372 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:49.799343    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-67184f2a-59c88bbd6c-j769f"]
Oct 30 22:15:49.802419 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:15:49.802375    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3129fc82_b397_4f74_a39c_f45e123c4ad1.slice/crio-6810aa0c8d05d09dabc480c192a42d038116c7f7714d1a377792f91628273fe6 WatchSource:0}: Error finding container 6810aa0c8d05d09dabc480c192a42d038116c7f7714d1a377792f91628273fe6: Status 404 returned error can't find the container with id 6810aa0c8d05d09dabc480c192a42d038116c7f7714d1a377792f91628273fe6
Oct 30 22:15:50.431587 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:50.431550    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-67184f2a-59c88bbd6c-j769f" event={"ID":"3129fc82-b397-4f74-a39c-f45e123c4ad1","Type":"ContainerStarted","Data":"6810aa0c8d05d09dabc480c192a42d038116c7f7714d1a377792f91628273fe6"}
Oct 30 22:15:51.448194 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:51.448160    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-67184f2a-59c88bbd6c-j769f" event={"ID":"3129fc82-b397-4f74-a39c-f45e123c4ad1","Type":"ContainerStarted","Data":"0515a778d1710d751383c737c4c9975938989992058253db9c2c7bbfbf2ebad9"}
Oct 30 22:15:51.448759 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:51.448733    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-67184f2a-59c88bbd6c-j769f"
Oct 30 22:15:51.570505 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:51.570426    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-67184f2a-59c88bbd6c-j769f" podStartSLOduration=2.57195632 podStartE2EDuration="3.570395418s" totalImagesPullingTime="998.439098ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:51.568532734 +0000 UTC m=+1529.172840300" watchObservedRunningTime="2026-10-30 22:15:51.570395418 +0000 UTC m=+1529.174703012"
Oct 30 22:15:57.463721 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:15:57.463688    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-67184f2a-59c88bbd6c-j769f"
Oct 30 22:16:03.128710 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:16:03.128496    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-72b5472b-6557fcb676-2x9c8"]
Oct 30 22:16:03.134362 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:16:03.134339    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-72b5472b-6557fcb676-2x9c8"
Oct 30 22:16:03.184686 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:16:03.184662    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-72b5472b-serving-cert\""
Oct 30 22:16:03.236260 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:16:03.236227    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-72b5472b-6557fcb676-2x9c8"]
Oct 30 22:16:03.299004 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:16:03.298977    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f6b333b-fdc7-4327-88e7-43272ca7cc51-metrics-certs\") pod \"maas-api-e2e-ait-72b5472b-6557fcb676-2x9c8\" (UID: \"5f6b333b-fdc7-4327-88e7-43272ca7cc51\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-72b5472b-6557fcb676-2x9c8"
Oct 30 22:16:03.299199 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:16:03.299009    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f6b333b-fdc7-4327-88e7-43272ca7cc51-openshift-service-ca\") pod \"maas-api-e2e-ait-72b5472b-6557fcb676-2x9c8\" (UID: \"5f6b333b-fdc7-4327-88e7-43272ca7cc51\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-72b5472b-6557fcb676-2x9c8"
Oct 30 22:16:03.299199 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:16:03.299164    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxzdp\" (UniqueName: \"kubernetes.io/projected/5f6b333b-fdc7-4327-88e7-43272ca7cc51-kube-api-access-cxzdp\") pod \"maas-api-e2e-ait-72b5472b-6557fcb676-2x9c8\" (UID: \"5f6b333b-fdc7-4327-88e7-43272ca7cc51\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-72b5472b-6557fcb676-2x9c8"
Oct 30 22:16:03.299304 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:16:03.299220    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f6b333b-fdc7-4327-88e7-43272ca7cc51-maas-api-tls\") pod \"maas-api-e2e-ait-72b5472b-6557fcb676-2x9c8\" (UID: \"5f6b333b-fdc7-4327-88e7-43272ca7cc51\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-72b5472b-6557fcb676-2x9c8"
Oct 30 22:16:03.400743 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:16:03.400659    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f6b333b-fdc7-4327-88e7-43272ca7cc51-metrics-certs\") pod \"maas-api-e2e-ait-72b5472b-6557fcb676-2x9c8\" (UID: \"5f6b333b-fdc7-4327-88e7-43272ca7cc51\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-72b5472b-6557fcb676-2x9c8"
Oct 30 22:16:03.400743 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:16:03.400702    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f6b333b-fdc7-4327-88e7-43272ca7cc51-openshift-service-ca\") pod \"maas-api-e2e-ait-72b5472b-6557fcb676-2x9c8\" (UID: \"5f6b333b-fdc7-4327-88e7-43272ca7cc51\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-72b5472b-6557fcb676-2x9c8"
Oct 30 22:16:03.400963 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:16:03.400803    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxzdp\" (UniqueName: \"kubernetes.io/projected/5f6b333b-fdc7-4327-88e7-43272ca7cc51-kube-api-access-cxzdp\") pod \"maas-api-e2e-ait-72b5472b-6557fcb676-2x9c8\" (UID: \"5f6b333b-fdc7-4327-88e7-43272ca7cc51\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-72b5472b-6557fcb676-2x9c8"
Oct 30 22:16:03.400963 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:16:03.400844    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f6b333b-fdc7-4327-88e7-43272ca7cc51-maas-api-tls\") pod \"maas-api-e2e-ait-72b5472b-6557fcb676-2x9c8\" (UID: \"5f6b333b-fdc7-4327-88e7-43272ca7cc51\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-72b5472b-6557fcb676-2x9c8"
Oct 30 22:16:03.401456 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:16:03.401424    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f6b333b-fdc7-4327-88e7-43272ca7cc51-openshift-service-ca\") pod \"maas-api-e2e-ait-72b5472b-6557fcb676-2x9c8\" (UID: \"5f6b333b-fdc7-4327-88e7-43272ca7cc51\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-72b5472b-6557fcb676-2x9c8"
Oct 30 22:16:03.403296 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:16:03.403274    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f6b333b-fdc7-4327-88e7-43272ca7cc51-maas-api-tls\") pod \"maas-api-e2e-ait-72b5472b-6557fcb676-2x9c8\" (UID: \"5f6b333b-fdc7-4327-88e7-43272ca7cc51\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-72b5472b-6557fcb676-2x9c8"
Oct 30 22:16:03.403535 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:16:03.403516    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f6b333b-fdc7-4327-88e7-43272ca7cc51-metrics-certs\") pod \"maas-api-e2e-ait-72b5472b-6557fcb676-2x9c8\" (UID: \"5f6b333b-fdc7-4327-88e7-43272ca7cc51\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-72b5472b-6557fcb676-2x9c8"
Oct 30 22:16:03.459733 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:16:03.459700    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxzdp\" (UniqueName: \"kubernetes.io/projected/5f6b333b-fdc7-4327-88e7-43272ca7cc51-kube-api-access-cxzdp\") pod \"maas-api-e2e-ait-72b5472b-6557fcb676-2x9c8\" (UID: \"5f6b333b-fdc7-4327-88e7-43272ca7cc51\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-72b5472b-6557fcb676-2x9c8"
Oct 30 22:16:03.748370 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:16:03.748284    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-72b5472b-6557fcb676-2x9c8"
Oct 30 22:16:04.133081 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:16:04.132997    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f6b333b_fdc7_4327_88e7_43272ca7cc51.slice/crio-5a2044ce72d786f357f0b42a54f887184460fe67606df5d41bf3ba464fefb13e WatchSource:0}: Error finding container 5a2044ce72d786f357f0b42a54f887184460fe67606df5d41bf3ba464fefb13e: Status 404 returned error can't find the container with id 5a2044ce72d786f357f0b42a54f887184460fe67606df5d41bf3ba464fefb13e
Oct 30 22:16:04.134776 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:16:04.134753    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-72b5472b-6557fcb676-2x9c8"]
Oct 30 22:16:04.577444 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:16:04.577340    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-72b5472b-6557fcb676-2x9c8" event={"ID":"5f6b333b-fdc7-4327-88e7-43272ca7cc51","Type":"ContainerStarted","Data":"5a2044ce72d786f357f0b42a54f887184460fe67606df5d41bf3ba464fefb13e"}
Oct 30 22:16:05.587350 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:16:05.587312    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-72b5472b-6557fcb676-2x9c8" event={"ID":"5f6b333b-fdc7-4327-88e7-43272ca7cc51","Type":"ContainerStarted","Data":"3672ce24469d34b98e2a26086196ac9f07c67598b574197664b157e87ea3d19e"}
Oct 30 22:16:05.587884 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:16:05.587396    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-72b5472b-6557fcb676-2x9c8"
Oct 30 22:16:05.708868 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:16:05.708812    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-72b5472b-6557fcb676-2x9c8" podStartSLOduration=1.8467661450000001 podStartE2EDuration="2.708797343s" totalImagesPullingTime="862.031198ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:16:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:16:05.703746795 +0000 UTC m=+1543.308054349" watchObservedRunningTime="2026-10-30 22:16:05.708797343 +0000 UTC m=+1543.313104924"
Oct 30 22:16:10.573896 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:16:10.573849    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 22:16:11.600275 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:16:11.600242    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-72b5472b-6557fcb676-2x9c8"
Oct 30 22:16:22.957015 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:16:22.956893    5678 scope.go:122] "RemoveContainer" containerID="713bf6c9cb2819553273cfe492307b86dd6fcb41e27ac925177c78a7cdaf340e"
Oct 30 22:16:27.155436 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:16:27.153167    5678 patch_prober.go:29] interesting pod/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-f2qtx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:16:27.155436 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:16:27.153245    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-f2qtx" podUID="95f62fb3-9a67-4bca-9483-8aedadf82cd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:16:27.216423 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:16:27.216001    5678 patch_prober.go:29] interesting pod/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-vzfdw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:16:27.216423 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:16:27.216113    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-vzfdw" podUID="24750677-9d8f-46f4-9f0e-acc3d230fb8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:16:27.278427 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:16:27.278204    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779f32/premium-w0-779f32-kserve-67978797c8-jpkvl" podUID="40138fb9-9a75-4458-a351-11eb61c4a1e3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:16:27.300062 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:16:27.299693    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7bb1c2/distinct-w3-7bb1c2-kserve-8d695596c-8zgtw" podUID="36f0a6a2-54be-49bf-9202-502960cbb134" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded"
Oct 30 22:16:27.307543 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:16:27.307506    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-65e59f/distinct-2-w4-65e59f-kserve-689566757d-bf98q" podUID="64e66fdb-056e-4086-a85a-8c02613584a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:16:27.312502 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:16:27.312466    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb0b93/distinct-2-w6-fb0b93-kserve-7d65bb4698-m6tkh" podUID="6fd276ef-522b-4bb4-b101-0a9c3fe9ecd3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:16:27.313003 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:16:27.312689    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-65e59f/premium-w4-65e59f-kserve-6dc49468f8-rlzhx" podUID="8590a783-eb24-4759-ad74-1369f36aa9d2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:16:27.817725 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:16:27.817677    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-65e59f/simulator-w4-65e59f-kserve-85698bcd47-przch" podUID="ee5e843b-17be-4623-8c95-dc07c8b7f506" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:16:27.972644 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:16:27.972572    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9ndd9" podUID="81bd6415-f42f-446b-a64e-956f794b3956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:16:28.277426 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:16:28.259524    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7bb1c2/simulator-w3-7bb1c2-kserve-675645869c-vcwqv" podUID="1fce9daf-68c9-41d0-825b-2012bb93c8ba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:16:28.307483 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:16:28.279775    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7bb1c2/premium-w3-7bb1c2-kserve-7dbf875d88-dqxn5" podUID="a58c911f-e2eb-4820-a6b6-24afe78b5fd4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:16:28.332802 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:16:28.327524    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-65e59f/premium-w4-65e59f-kserve-6dc49468f8-rlzhx" podUID="8590a783-eb24-4759-ad74-1369f36aa9d2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:16:28.332802 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:16:28.327830    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb0b93/distinct-2-w6-fb0b93-kserve-7d65bb4698-m6tkh" podUID="6fd276ef-522b-4bb4-b101-0a9c3fe9ecd3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": context deadline exceeded"
Oct 30 22:16:28.332802 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:16:28.328047    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-65e59f/premium-w4-65e59f-kserve-6dc49468f8-rlzhx"
Oct 30 22:16:28.332802 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:16:28.328104    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-fb0b93/distinct-2-w6-fb0b93-kserve-7d65bb4698-m6tkh"
Oct 30 22:16:28.332802 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:16:28.330734    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb0b93/simulator-w6-fb0b93-kserve-55b784b479-588bn" podUID="951a6f7e-a22e-40f2-ace2-1ce5d3bc9b49" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:16:28.342945 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:16:28.340579    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-65e59f/distinct-2-w4-65e59f-kserve-689566757d-bf98q" podUID="64e66fdb-056e-4086-a85a-8c02613584a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:16:28.342945 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:16:28.342100    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-65e59f/distinct-2-w4-65e59f-kserve-689566757d-bf98q"
Oct 30 22:16:28.354592 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:16:28.354483    5678 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-779f32/premium-w0-779f32-kserve-67978797c8-jpkvl" podUID="40138fb9-9a75-4458-a351-11eb61c4a1e3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:16:28.422447 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:16:28.420166    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-65e59f/premium-w4-65e59f-kserve-6dc49468f8-rlzhx"
Oct 30 22:16:28.422447 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:16:28.420662    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a00d9b/distinct-2-w1-a00d9b-kserve-68fd95957c-dmgnb" podUID="23a81fda-f7e3-4f91-8d96-9f19c7de6e09" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:16:28.428338 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:16:28.427869    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-65e59f/distinct-2-w4-65e59f-kserve-689566757d-bf98q"
Oct 30 22:16:28.801502 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:16:28.801469    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-fb0b93/distinct-2-w6-fb0b93-kserve-7d65bb4698-m6tkh"
Oct 30 22:16:37.008300 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:16:37.008267    5678 generic.go:356] "Generic (PLEG): container finished" podID="313acbe0-dc5c-48cb-92d8-972092227317" containerID="141a472a7e05afdc52b72cd370f0ad46fc2206ebc9a7bae42c951e83738cc0fb" exitCode=0
Oct 30 22:16:37.008735 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:16:37.008342    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ff716ef6-openshift-default-5bbc788795-f68kp" event={"ID":"313acbe0-dc5c-48cb-92d8-972092227317","Type":"ContainerDied","Data":"141a472a7e05afdc52b72cd370f0ad46fc2206ebc9a7bae42c951e83738cc0fb"}
Oct 30 22:16:37.008779 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:16:37.008769    5678 scope.go:122] "RemoveContainer" containerID="141a472a7e05afdc52b72cd370f0ad46fc2206ebc9a7bae42c951e83738cc0fb"
Oct 30 22:16:37.010515 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:16:37.010485    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:16:37.010657 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:16:37.010573    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:16:37.010657 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:16:37.010616    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:16:38.017767 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:16:38.017733    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ff716ef6-openshift-default-5bbc788795-f68kp" event={"ID":"313acbe0-dc5c-48cb-92d8-972092227317","Type":"ContainerStarted","Data":"99163503e6d645c29cca4298bd14c39826e7f00ab72ec6e5efa30a7d73208cf4"}
Oct 30 22:16:38.848311 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:16:38.848267    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-ff716ef6-openshift-default-5bbc788795-f68kp"
Oct 30 22:16:38.849872 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:16:38.849844    5678 patch_prober.go:29] interesting pod/e2e-ait-ff716ef6-openshift-default-5bbc788795-f68kp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body=
Oct 30 22:16:38.850019 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:16:38.849904    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ff716ef6-openshift-default-5bbc788795-f68kp" podUID="313acbe0-dc5c-48cb-92d8-972092227317" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused"
Oct 30 22:16:39.746695 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:16:39.746661    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-ff716ef6-openshift-default-5bbc788795-f68kp"
Oct 30 22:16:39.848753 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:16:39.848707    5678 patch_prober.go:29] interesting pod/e2e-ait-ff716ef6-openshift-default-5bbc788795-f68kp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body=
Oct 30 22:16:39.848950 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:16:39.848770    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ff716ef6-openshift-default-5bbc788795-f68kp" podUID="313acbe0-dc5c-48cb-92d8-972092227317" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused"
Oct 30 22:16:40.574452 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:16:40.574420    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 22:16:40.849121 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:16:40.849042    5678 patch_prober.go:29] interesting pod/e2e-ait-ff716ef6-openshift-default-5bbc788795-f68kp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body=
Oct 30 22:16:40.849121 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:16:40.849106    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ff716ef6-openshift-default-5bbc788795-f68kp" podUID="313acbe0-dc5c-48cb-92d8-972092227317" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused"
Oct 30 22:16:41.848751 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:16:41.848717    5678 patch_prober.go:29] interesting pod/e2e-ait-ff716ef6-openshift-default-5bbc788795-f68kp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body=
Oct 30 22:16:41.848927 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:16:41.848780    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ff716ef6-openshift-default-5bbc788795-f68kp" podUID="313acbe0-dc5c-48cb-92d8-972092227317" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused"
Oct 30 22:16:42.848642 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:16:42.848606    5678 patch_prober.go:29] interesting pod/e2e-ait-ff716ef6-openshift-default-5bbc788795-f68kp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body=
Oct 30 22:16:42.849043 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:16:42.848656    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ff716ef6-openshift-default-5bbc788795-f68kp" podUID="313acbe0-dc5c-48cb-92d8-972092227317" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused"
Oct 30 22:16:43.852854 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:16:43.852824    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-ff716ef6-openshift-default-5bbc788795-f68kp"
Oct 30 22:16:43.853883 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:16:43.853861    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-ff716ef6-openshift-default-5bbc788795-f68kp"
Oct 30 22:16:53.073956 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:16:53.073926    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbca0da1-tg22z"]
Oct 30 22:16:53.120329 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:16:53.120302    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbca0da1-tg22z"]
Oct 30 22:16:54.525595 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:16:54.525566    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9147be34-042d-49b5-9f75-2f2440a6dae9" path="/var/lib/kubelet/pods/9147be34-042d-49b5-9f75-2f2440a6dae9/volumes"
Oct 30 22:16:58.747460 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:16:58.747428    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-x6j2w"]
Oct 30 22:16:58.751510 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:16:58.751485    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-x6j2w"
Oct 30 22:16:58.871267 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:16:58.871233    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-x6j2w"]
Oct 30 22:16:58.915564 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:16:58.915525    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea5fcd11-31d9-4836-92c3-00952b348844-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-x6j2w\" (UID: \"ea5fcd11-31d9-4836-92c3-00952b348844\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-x6j2w"
Oct 30 22:16:58.915724 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:16:58.915691    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea5fcd11-31d9-4836-92c3-00952b348844-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-x6j2w\" (UID: \"ea5fcd11-31d9-4836-92c3-00952b348844\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-x6j2w"
Oct 30 22:16:58.915724 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:16:58.915719    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea5fcd11-31d9-4836-92c3-00952b348844-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-x6j2w\" (UID: \"ea5fcd11-31d9-4836-92c3-00952b348844\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-x6j2w"
Oct 30 22:16:58.915912 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:16:58.915891    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea5fcd11-31d9-4836-92c3-00952b348844-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-x6j2w\" (UID: \"ea5fcd11-31d9-4836-92c3-00952b348844\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-x6j2w"
Oct 30 22:16:58.915952 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:16:58.915930    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea5fcd11-31d9-4836-92c3-00952b348844-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-x6j2w\" (UID: \"ea5fcd11-31d9-4836-92c3-00952b348844\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-x6j2w"
Oct 30 22:16:58.916036 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:16:58.916014    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea5fcd11-31d9-4836-92c3-00952b348844-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-x6j2w\" (UID: \"ea5fcd11-31d9-4836-92c3-00952b348844\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-x6j2w"
Oct 30 22:16:58.916084 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:16:58.916068    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mp69h\" (UniqueName: \"kubernetes.io/projected/ea5fcd11-31d9-4836-92c3-00952b348844-kube-api-access-mp69h\") pod \"maas-default-gateway-openshift-default-85544d4cd7-x6j2w\" (UID: \"ea5fcd11-31d9-4836-92c3-00952b348844\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-x6j2w"
Oct 30 22:16:58.916201 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:16:58.916188    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea5fcd11-31d9-4836-92c3-00952b348844-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-x6j2w\" (UID: \"ea5fcd11-31d9-4836-92c3-00952b348844\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-x6j2w"
Oct 30 22:16:58.916239 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:16:58.916222    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea5fcd11-31d9-4836-92c3-00952b348844-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-x6j2w\" (UID: \"ea5fcd11-31d9-4836-92c3-00952b348844\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-x6j2w"
Oct 30 22:16:59.017820 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:16:59.017741    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea5fcd11-31d9-4836-92c3-00952b348844-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-x6j2w\" (UID: \"ea5fcd11-31d9-4836-92c3-00952b348844\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-x6j2w"
Oct 30 22:16:59.017820 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:16:59.017787    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea5fcd11-31d9-4836-92c3-00952b348844-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-x6j2w\" (UID: \"ea5fcd11-31d9-4836-92c3-00952b348844\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-x6j2w"
Oct 30 22:16:59.018047 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:16:59.017872    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea5fcd11-31d9-4836-92c3-00952b348844-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-x6j2w\" (UID: \"ea5fcd11-31d9-4836-92c3-00952b348844\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-x6j2w"
Oct 30 22:16:59.018047 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:16:59.017951    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea5fcd11-31d9-4836-92c3-00952b348844-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-x6j2w\" (UID: \"ea5fcd11-31d9-4836-92c3-00952b348844\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-x6j2w"
Oct 30 22:16:59.018167 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:16:59.018070    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea5fcd11-31d9-4836-92c3-00952b348844-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-x6j2w\" (UID: \"ea5fcd11-31d9-4836-92c3-00952b348844\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-x6j2w"
Oct 30 22:16:59.018225 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:16:59.018195    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea5fcd11-31d9-4836-92c3-00952b348844-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-x6j2w\" (UID: \"ea5fcd11-31d9-4836-92c3-00952b348844\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-x6j2w"
Oct 30 22:16:59.018284 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:16:59.018219    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea5fcd11-31d9-4836-92c3-00952b348844-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-x6j2w\" (UID: \"ea5fcd11-31d9-4836-92c3-00952b348844\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-x6j2w"
Oct 30 22:16:59.018284 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:16:59.018223    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea5fcd11-31d9-4836-92c3-00952b348844-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-x6j2w\" (UID: \"ea5fcd11-31d9-4836-92c3-00952b348844\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-x6j2w"
Oct 30 22:16:59.018284 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:16:59.018264    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea5fcd11-31d9-4836-92c3-00952b348844-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-x6j2w\" (UID: \"ea5fcd11-31d9-4836-92c3-00952b348844\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-x6j2w"
Oct 30 22:16:59.018503 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:16:59.018293    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea5fcd11-31d9-4836-92c3-00952b348844-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-x6j2w\" (UID: \"ea5fcd11-31d9-4836-92c3-00952b348844\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-x6j2w"
Oct 30 22:16:59.018503 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:16:59.018297    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea5fcd11-31d9-4836-92c3-00952b348844-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-x6j2w\" (UID: \"ea5fcd11-31d9-4836-92c3-00952b348844\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-x6j2w"
Oct 30 22:16:59.018503 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:16:59.018352    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mp69h\" (UniqueName: \"kubernetes.io/projected/ea5fcd11-31d9-4836-92c3-00952b348844-kube-api-access-mp69h\") pod \"maas-default-gateway-openshift-default-85544d4cd7-x6j2w\" (UID: \"ea5fcd11-31d9-4836-92c3-00952b348844\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-x6j2w"
Oct 30 22:16:59.018663 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:16:59.018601    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea5fcd11-31d9-4836-92c3-00952b348844-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-x6j2w\" (UID: \"ea5fcd11-31d9-4836-92c3-00952b348844\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-x6j2w"
Oct 30 22:16:59.018871 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:16:59.018843    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea5fcd11-31d9-4836-92c3-00952b348844-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-x6j2w\" (UID: \"ea5fcd11-31d9-4836-92c3-00952b348844\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-x6j2w"
Oct 30 22:16:59.020345 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:16:59.020322    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea5fcd11-31d9-4836-92c3-00952b348844-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-x6j2w\" (UID: \"ea5fcd11-31d9-4836-92c3-00952b348844\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-x6j2w"
Oct 30 22:16:59.020465 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:16:59.020381    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea5fcd11-31d9-4836-92c3-00952b348844-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-x6j2w\" (UID: \"ea5fcd11-31d9-4836-92c3-00952b348844\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-x6j2w"
Oct 30 22:16:59.079637 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:16:59.079602    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea5fcd11-31d9-4836-92c3-00952b348844-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-x6j2w\" (UID: \"ea5fcd11-31d9-4836-92c3-00952b348844\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-x6j2w"
Oct 30 22:16:59.079756 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:16:59.079694    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mp69h\" (UniqueName: \"kubernetes.io/projected/ea5fcd11-31d9-4836-92c3-00952b348844-kube-api-access-mp69h\") pod \"maas-default-gateway-openshift-default-85544d4cd7-x6j2w\" (UID: \"ea5fcd11-31d9-4836-92c3-00952b348844\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-x6j2w"
Oct 30 22:16:59.365353 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:16:59.365320    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-x6j2w"
Oct 30 22:16:59.816229 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:16:59.816146    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-x6j2w"]
Oct 30 22:16:59.849900 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:16:59.849873    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea5fcd11_31d9_4836_92c3_00952b348844.slice/crio-5cc3b2e3b5c564418e6878cf2c41363c99529405726380861281c898438a1af1 WatchSource:0}: Error finding container 5cc3b2e3b5c564418e6878cf2c41363c99529405726380861281c898438a1af1: Status 404 returned error can't find the container with id 5cc3b2e3b5c564418e6878cf2c41363c99529405726380861281c898438a1af1
Oct 30 22:16:59.852539 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:16:59.852506    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:16:59.852605 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:16:59.852579    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:16:59.852646 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:16:59.852606    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:17:00.224819 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:00.224783    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-x6j2w" event={"ID":"ea5fcd11-31d9-4836-92c3-00952b348844","Type":"ContainerStarted","Data":"dc67d0e32a6d45b2b0e6108b6394c96b0c5562fadd64fb273ad675b6f82d0152"}
Oct 30 22:17:00.224996 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:00.224840    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-x6j2w" event={"ID":"ea5fcd11-31d9-4836-92c3-00952b348844","Type":"ContainerStarted","Data":"5cc3b2e3b5c564418e6878cf2c41363c99529405726380861281c898438a1af1"}
Oct 30 22:17:00.366388 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:00.366347    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-x6j2w"
Oct 30 22:17:00.368688 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:00.368660    5678 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-x6j2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body=
Oct 30 22:17:00.368825 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:00.368724    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-x6j2w" podUID="ea5fcd11-31d9-4836-92c3-00952b348844" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused"
Oct 30 22:17:00.399153 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:00.399104    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-x6j2w" podStartSLOduration=2.399089435 podStartE2EDuration="2.399089435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:16:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:17:00.392907848 +0000 UTC m=+1597.997215406" watchObservedRunningTime="2026-10-30 22:17:00.399089435 +0000 UTC m=+1598.003396987"
Oct 30 22:17:01.365997 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:01.365960    5678 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-x6j2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body=
Oct 30 22:17:01.366464 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:01.366019    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-x6j2w" podUID="ea5fcd11-31d9-4836-92c3-00952b348844" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused"
Oct 30 22:17:02.366078 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:02.366029    5678 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-x6j2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body=
Oct 30 22:17:02.366500 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:02.366100    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-x6j2w" podUID="ea5fcd11-31d9-4836-92c3-00952b348844" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused"
Oct 30 22:17:03.365974 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:03.365938    5678 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-x6j2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body=
Oct 30 22:17:03.366153 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:03.366000    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-x6j2w" podUID="ea5fcd11-31d9-4836-92c3-00952b348844" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused"
Oct 30 22:17:04.365913 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:04.365873    5678 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-x6j2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body=
Oct 30 22:17:04.366099 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:04.365936    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-x6j2w" podUID="ea5fcd11-31d9-4836-92c3-00952b348844" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused"
Oct 30 22:17:05.366124 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:05.366094    5678 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-x6j2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body=
Oct 30 22:17:05.366655 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:05.366168    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-x6j2w" podUID="ea5fcd11-31d9-4836-92c3-00952b348844" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused"
Oct 30 22:17:06.366045 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:06.366007    5678 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-x6j2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body=
Oct 30 22:17:06.366228 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:06.366070    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-x6j2w" podUID="ea5fcd11-31d9-4836-92c3-00952b348844" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused"
Oct 30 22:17:07.366034 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:07.365991    5678 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-x6j2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body=
Oct 30 22:17:07.366214 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:07.366065    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-x6j2w" podUID="ea5fcd11-31d9-4836-92c3-00952b348844" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused"
Oct 30 22:17:08.483124 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:08.483092    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-x6j2w"
Oct 30 22:17:08.484371 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:08.484346    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-x6j2w"
Oct 30 22:17:08.485014 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:08.484998    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-x6j2w"
Oct 30 22:17:10.574336 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:17:10.574300    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 22:17:13.867903 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:13.867868    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xq4tp"]
Oct 30 22:17:13.871997 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:13.871972    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xq4tp"
Oct 30 22:17:13.909217 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:13.909180    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjrwz\" (UniqueName: \"kubernetes.io/projected/b2fc93b9-e77d-4306-acec-85ea06f5c4be-kube-api-access-mjrwz\") pod \"maas-default-gateway-openshift-default-85544d4cd7-xq4tp\" (UID: \"b2fc93b9-e77d-4306-acec-85ea06f5c4be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xq4tp"
Oct 30 22:17:13.909376 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:13.909241    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2fc93b9-e77d-4306-acec-85ea06f5c4be-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-xq4tp\" (UID: \"b2fc93b9-e77d-4306-acec-85ea06f5c4be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xq4tp"
Oct 30 22:17:13.909478 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:13.909455    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2fc93b9-e77d-4306-acec-85ea06f5c4be-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-xq4tp\" (UID: \"b2fc93b9-e77d-4306-acec-85ea06f5c4be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xq4tp"
Oct 30 22:17:13.909521 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:13.909508    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2fc93b9-e77d-4306-acec-85ea06f5c4be-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-xq4tp\" (UID: \"b2fc93b9-e77d-4306-acec-85ea06f5c4be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xq4tp"
Oct 30 22:17:13.909564 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:13.909549    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2fc93b9-e77d-4306-acec-85ea06f5c4be-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-xq4tp\" (UID: \"b2fc93b9-e77d-4306-acec-85ea06f5c4be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xq4tp"
Oct 30 22:17:13.909610 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:13.909599    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2fc93b9-e77d-4306-acec-85ea06f5c4be-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-xq4tp\" (UID: \"b2fc93b9-e77d-4306-acec-85ea06f5c4be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xq4tp"
Oct 30 22:17:13.909671 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:13.909656    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2fc93b9-e77d-4306-acec-85ea06f5c4be-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-xq4tp\" (UID: \"b2fc93b9-e77d-4306-acec-85ea06f5c4be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xq4tp"
Oct 30 22:17:13.909709 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:13.909699    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2fc93b9-e77d-4306-acec-85ea06f5c4be-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-xq4tp\" (UID: \"b2fc93b9-e77d-4306-acec-85ea06f5c4be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xq4tp"
Oct 30 22:17:13.909743 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:13.909717    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2fc93b9-e77d-4306-acec-85ea06f5c4be-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-xq4tp\" (UID: \"b2fc93b9-e77d-4306-acec-85ea06f5c4be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xq4tp"
Oct 30 22:17:13.983179 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:13.983145    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xq4tp"]
Oct 30 22:17:14.011552 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:14.011490    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjrwz\" (UniqueName: \"kubernetes.io/projected/b2fc93b9-e77d-4306-acec-85ea06f5c4be-kube-api-access-mjrwz\") pod \"maas-default-gateway-openshift-default-85544d4cd7-xq4tp\" (UID: \"b2fc93b9-e77d-4306-acec-85ea06f5c4be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xq4tp"
Oct 30 22:17:14.011756 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:14.011586    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2fc93b9-e77d-4306-acec-85ea06f5c4be-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-xq4tp\" (UID: \"b2fc93b9-e77d-4306-acec-85ea06f5c4be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xq4tp"
Oct 30 22:17:14.011827 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:14.011763    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2fc93b9-e77d-4306-acec-85ea06f5c4be-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-xq4tp\" (UID: \"b2fc93b9-e77d-4306-acec-85ea06f5c4be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xq4tp"
Oct 30 22:17:14.011884 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:14.011824    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2fc93b9-e77d-4306-acec-85ea06f5c4be-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-xq4tp\" (UID: \"b2fc93b9-e77d-4306-acec-85ea06f5c4be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xq4tp"
Oct 30 22:17:14.011884 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:14.011853    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2fc93b9-e77d-4306-acec-85ea06f5c4be-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-xq4tp\" (UID: \"b2fc93b9-e77d-4306-acec-85ea06f5c4be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xq4tp"
Oct 30 22:17:14.012000 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:14.011897    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2fc93b9-e77d-4306-acec-85ea06f5c4be-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-xq4tp\" (UID: \"b2fc93b9-e77d-4306-acec-85ea06f5c4be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xq4tp"
Oct 30 22:17:14.012000 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:14.011923    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2fc93b9-e77d-4306-acec-85ea06f5c4be-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-xq4tp\" (UID: \"b2fc93b9-e77d-4306-acec-85ea06f5c4be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xq4tp"
Oct 30 22:17:14.012000 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:14.011977    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2fc93b9-e77d-4306-acec-85ea06f5c4be-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-xq4tp\" (UID: \"b2fc93b9-e77d-4306-acec-85ea06f5c4be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xq4tp"
Oct 30 22:17:14.012163 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:14.012002    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2fc93b9-e77d-4306-acec-85ea06f5c4be-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-xq4tp\" (UID: \"b2fc93b9-e77d-4306-acec-85ea06f5c4be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xq4tp"
Oct 30 22:17:14.012163 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:14.012088    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2fc93b9-e77d-4306-acec-85ea06f5c4be-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-xq4tp\" (UID: \"b2fc93b9-e77d-4306-acec-85ea06f5c4be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xq4tp"
Oct 30 22:17:14.012270 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:14.012191    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2fc93b9-e77d-4306-acec-85ea06f5c4be-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-xq4tp\" (UID: \"b2fc93b9-e77d-4306-acec-85ea06f5c4be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xq4tp"
Oct 30 22:17:14.012491 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:14.012466    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2fc93b9-e77d-4306-acec-85ea06f5c4be-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-xq4tp\" (UID: \"b2fc93b9-e77d-4306-acec-85ea06f5c4be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xq4tp"
Oct 30 22:17:14.012622 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:14.012541    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2fc93b9-e77d-4306-acec-85ea06f5c4be-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-xq4tp\" (UID: \"b2fc93b9-e77d-4306-acec-85ea06f5c4be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xq4tp"
Oct 30 22:17:14.013240 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:14.013212    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2fc93b9-e77d-4306-acec-85ea06f5c4be-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-xq4tp\" (UID: \"b2fc93b9-e77d-4306-acec-85ea06f5c4be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xq4tp"
Oct 30 22:17:14.014958 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:14.014930    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2fc93b9-e77d-4306-acec-85ea06f5c4be-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-xq4tp\" (UID: \"b2fc93b9-e77d-4306-acec-85ea06f5c4be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xq4tp"
Oct 30 22:17:14.015067 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:14.014952    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2fc93b9-e77d-4306-acec-85ea06f5c4be-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-xq4tp\" (UID: \"b2fc93b9-e77d-4306-acec-85ea06f5c4be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xq4tp"
Oct 30 22:17:14.070235 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:14.070208    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2fc93b9-e77d-4306-acec-85ea06f5c4be-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-xq4tp\" (UID: \"b2fc93b9-e77d-4306-acec-85ea06f5c4be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xq4tp"
Oct 30 22:17:14.070378 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:14.070355    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjrwz\" (UniqueName: \"kubernetes.io/projected/b2fc93b9-e77d-4306-acec-85ea06f5c4be-kube-api-access-mjrwz\") pod \"maas-default-gateway-openshift-default-85544d4cd7-xq4tp\" (UID: \"b2fc93b9-e77d-4306-acec-85ea06f5c4be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xq4tp"
Oct 30 22:17:14.186149 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:14.186063    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xq4tp"
Oct 30 22:17:14.634386 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:17:14.634352    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2fc93b9_e77d_4306_acec_85ea06f5c4be.slice/crio-5e55ed36e69b6dfb8ce0574e94ba9ee7cbf5864a00e7586c133cf2cfee0b1e71 WatchSource:0}: Error finding container 5e55ed36e69b6dfb8ce0574e94ba9ee7cbf5864a00e7586c133cf2cfee0b1e71: Status 404 returned error can't find the container with id 5e55ed36e69b6dfb8ce0574e94ba9ee7cbf5864a00e7586c133cf2cfee0b1e71
Oct 30 22:17:14.634559 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:14.634528    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xq4tp"]
Oct 30 22:17:14.636865 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:14.636834    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:17:14.636954 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:14.636897    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:17:14.636954 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:14.636926    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:17:15.399297 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:15.399253    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xq4tp" event={"ID":"b2fc93b9-e77d-4306-acec-85ea06f5c4be","Type":"ContainerStarted","Data":"e0b2c8c99a7b725e5c0ea317d2bd9569b40e2138e4ce49269cf5c7e79ba9b2a6"}
Oct 30 22:17:15.399793 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:15.399341    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xq4tp" event={"ID":"b2fc93b9-e77d-4306-acec-85ea06f5c4be","Type":"ContainerStarted","Data":"5e55ed36e69b6dfb8ce0574e94ba9ee7cbf5864a00e7586c133cf2cfee0b1e71"}
Oct 30 22:17:15.607423 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:15.602926    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xq4tp" podStartSLOduration=2.602905573 podStartE2EDuration="2.602905573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:17:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:17:15.58792256 +0000 UTC m=+1613.192230114" watchObservedRunningTime="2026-10-30 22:17:15.602905573 +0000 UTC m=+1613.207213127"
Oct 30 22:17:16.186284 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:16.186248    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xq4tp"
Oct 30 22:17:16.188241 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:16.188211    5678 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-xq4tp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body=
Oct 30 22:17:16.188362 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:16.188285    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xq4tp" podUID="b2fc93b9-e77d-4306-acec-85ea06f5c4be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused"
Oct 30 22:17:17.187001 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:17.186953    5678 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-xq4tp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body=
Oct 30 22:17:17.187583 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:17.187037    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xq4tp" podUID="b2fc93b9-e77d-4306-acec-85ea06f5c4be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused"
Oct 30 22:17:18.190424 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:18.189854    5678 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-xq4tp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body=
Oct 30 22:17:18.190424 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:18.189917    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xq4tp" podUID="b2fc93b9-e77d-4306-acec-85ea06f5c4be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused"
Oct 30 22:17:18.272278 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:18.271488    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779f32/premium-w0-779f32-kserve-67978797c8-jpkvl" podUID="40138fb9-9a75-4458-a351-11eb61c4a1e3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Oct 30 22:17:18.273368 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:18.272580    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7bb1c2/premium-w3-7bb1c2-kserve-7dbf875d88-dqxn5" podUID="a58c911f-e2eb-4820-a6b6-24afe78b5fd4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:17:18.285232 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:18.284473    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7bb1c2/distinct-w3-7bb1c2-kserve-8d695596c-8zgtw" podUID="36f0a6a2-54be-49bf-9202-502960cbb134" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:17:18.337605 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:18.337552    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb0b93/simulator-w6-fb0b93-kserve-55b784b479-588bn" podUID="951a6f7e-a22e-40f2-ace2-1ce5d3bc9b49" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:17:18.382559 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:18.382496    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a00d9b/distinct-2-w1-a00d9b-kserve-68fd95957c-dmgnb" podUID="23a81fda-f7e3-4f91-8d96-9f19c7de6e09" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:17:18.426159 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:18.426076    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-65e59f/premium-w4-65e59f-kserve-6dc49468f8-rlzhx" podUID="8590a783-eb24-4759-ad74-1369f36aa9d2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:17:18.431293 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:18.430486    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-65e59f/distinct-2-w4-65e59f-kserve-689566757d-bf98q" podUID="64e66fdb-056e-4086-a85a-8c02613584a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:17:18.809499 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:18.807497    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-65e59f/simulator-w4-65e59f-kserve-85698bcd47-przch" podUID="ee5e843b-17be-4623-8c95-dc07c8b7f506" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:17:18.809499 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:18.807964    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb0b93/distinct-2-w6-fb0b93-kserve-7d65bb4698-m6tkh" podUID="6fd276ef-522b-4bb4-b101-0a9c3fe9ecd3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:17:18.963408 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:18.963348    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9ndd9" podUID="81bd6415-f42f-446b-a64e-956f794b3956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:17:19.008129 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:19.008082    5678 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-65e59f/distinct-2-w4-65e59f-kserve-689566757d-bf98q" podUID="64e66fdb-056e-4086-a85a-8c02613584a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/health\": context deadline exceeded"
Oct 30 22:17:19.200921 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:19.200885    5678 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-xq4tp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body=
Oct 30 22:17:19.201521 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:19.201479    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xq4tp" podUID="b2fc93b9-e77d-4306-acec-85ea06f5c4be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused"
Oct 30 22:17:19.250529 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:19.250480    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7bb1c2/simulator-w3-7bb1c2-kserve-675645869c-vcwqv" podUID="1fce9daf-68c9-41d0-825b-2012bb93c8ba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:17:19.273517 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:19.273473    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779f32/premium-w0-779f32-kserve-67978797c8-jpkvl" podUID="40138fb9-9a75-4458-a351-11eb61c4a1e3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:17:19.274152 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:19.274131    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-779f32/premium-w0-779f32-kserve-67978797c8-jpkvl"
Oct 30 22:17:19.276160 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:19.276131    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7bb1c2/premium-w3-7bb1c2-kserve-7dbf875d88-dqxn5" podUID="a58c911f-e2eb-4820-a6b6-24afe78b5fd4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:17:19.276597 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:19.276578    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-7bb1c2/premium-w3-7bb1c2-kserve-7dbf875d88-dqxn5"
Oct 30 22:17:19.303422 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:19.287476    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7bb1c2/distinct-w3-7bb1c2-kserve-8d695596c-8zgtw" podUID="36f0a6a2-54be-49bf-9202-502960cbb134" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:17:19.303422 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:19.287817    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-7bb1c2/distinct-w3-7bb1c2-kserve-8d695596c-8zgtw"
Oct 30 22:17:19.355246 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:19.355208    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb0b93/simulator-w6-fb0b93-kserve-55b784b479-588bn" podUID="951a6f7e-a22e-40f2-ace2-1ce5d3bc9b49" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 30 22:17:19.357247 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:19.357225    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-fb0b93/simulator-w6-fb0b93-kserve-55b784b479-588bn"
Oct 30 22:17:19.383515 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:19.383480    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a00d9b/distinct-2-w1-a00d9b-kserve-68fd95957c-dmgnb" podUID="23a81fda-f7e3-4f91-8d96-9f19c7de6e09" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:17:19.384063 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:19.384040    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a00d9b/distinct-2-w1-a00d9b-kserve-68fd95957c-dmgnb"
Oct 30 22:17:19.438426 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:19.435022    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-65e59f/premium-w4-65e59f-kserve-6dc49468f8-rlzhx" podUID="8590a783-eb24-4759-ad74-1369f36aa9d2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded"
Oct 30 22:17:19.438426 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:19.436490    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-65e59f/premium-w4-65e59f-kserve-6dc49468f8-rlzhx"
Oct 30 22:17:19.439629 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:19.438729    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-65e59f/distinct-2-w4-65e59f-kserve-689566757d-bf98q" podUID="64e66fdb-056e-4086-a85a-8c02613584a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:17:19.439629 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:19.439074    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-65e59f/distinct-2-w4-65e59f-kserve-689566757d-bf98q"
Oct 30 22:17:19.836458 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:19.809492    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb0b93/distinct-2-w6-fb0b93-kserve-7d65bb4698-m6tkh" podUID="6fd276ef-522b-4bb4-b101-0a9c3fe9ecd3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:17:19.836458 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:19.809894    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-fb0b93/distinct-2-w6-fb0b93-kserve-7d65bb4698-m6tkh"
Oct 30 22:17:19.980476 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:19.980435    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9ndd9" podUID="81bd6415-f42f-446b-a64e-956f794b3956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:17:20.000433 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:19.999534    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9ndd9"
Oct 30 22:17:20.076549 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:20.076499    5678 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-7bb1c2/premium-w3-7bb1c2-kserve-7dbf875d88-dqxn5" podUID="a58c911f-e2eb-4820-a6b6-24afe78b5fd4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:17:20.329539 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:20.329487    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7bb1c2/distinct-w3-7bb1c2-kserve-8d695596c-8zgtw" podUID="36f0a6a2-54be-49bf-9202-502960cbb134" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:17:20.381698 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:20.352176    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7bb1c2/premium-w3-7bb1c2-kserve-7dbf875d88-dqxn5" podUID="a58c911f-e2eb-4820-a6b6-24afe78b5fd4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:17:20.381698 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:20.357051    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb0b93/simulator-w6-fb0b93-kserve-55b784b479-588bn" podUID="951a6f7e-a22e-40f2-ace2-1ce5d3bc9b49" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 30 22:17:20.381698 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:20.379524    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779f32/premium-w0-779f32-kserve-67978797c8-jpkvl" podUID="40138fb9-9a75-4458-a351-11eb61c4a1e3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:17:20.381698 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:20.381313    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7bb1c2/simulator-w3-7bb1c2-kserve-675645869c-vcwqv" podUID="1fce9daf-68c9-41d0-825b-2012bb93c8ba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Oct 30 22:17:20.381698 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:20.381572    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-7bb1c2/simulator-w3-7bb1c2-kserve-675645869c-vcwqv"
Oct 30 22:17:20.395542 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:20.384990    5678 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-xq4tp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body=
Oct 30 22:17:20.395542 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:20.385031    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xq4tp" podUID="b2fc93b9-e77d-4306-acec-85ea06f5c4be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused"
Oct 30 22:17:20.440043 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:20.426119    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a00d9b/distinct-2-w1-a00d9b-kserve-68fd95957c-dmgnb" podUID="23a81fda-f7e3-4f91-8d96-9f19c7de6e09" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded"
Oct 30 22:17:20.440043 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:20.438518    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-65e59f/premium-w4-65e59f-kserve-6dc49468f8-rlzhx" podUID="8590a783-eb24-4759-ad74-1369f36aa9d2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:17:20.448273 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:20.440828    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-65e59f/distinct-2-w4-65e59f-kserve-689566757d-bf98q" podUID="64e66fdb-056e-4086-a85a-8c02613584a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:17:20.776496 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:20.776466    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-7bb1c2/simulator-w3-7bb1c2-kserve-675645869c-vcwqv"
Oct 30 22:17:20.818994 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:20.818953    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb0b93/distinct-2-w6-fb0b93-kserve-7d65bb4698-m6tkh" podUID="6fd276ef-522b-4bb4-b101-0a9c3fe9ecd3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:17:20.840542 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:20.840499    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-65e59f/simulator-w4-65e59f-kserve-85698bcd47-przch" podUID="ee5e843b-17be-4623-8c95-dc07c8b7f506" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:17:20.898991 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:20.898962    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-65e59f/distinct-2-w4-65e59f-kserve-689566757d-bf98q"
Oct 30 22:17:20.975081 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:20.973630    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-7bb1c2/premium-w3-7bb1c2-kserve-7dbf875d88-dqxn5"
Oct 30 22:17:20.975081 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:20.973686    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-779f32/premium-w0-779f32-kserve-67978797c8-jpkvl"
Oct 30 22:17:20.981086 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:20.980820    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9ndd9" podUID="81bd6415-f42f-446b-a64e-956f794b3956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:17:21.007091 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:21.006467    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9ndd9"
Oct 30 22:17:21.277702 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:21.277574    5678 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-xq4tp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body=
Oct 30 22:17:21.277702 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:21.277642    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xq4tp" podUID="b2fc93b9-e77d-4306-acec-85ea06f5c4be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused"
Oct 30 22:17:21.363424 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:21.333709    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7bb1c2/distinct-w3-7bb1c2-kserve-8d695596c-8zgtw" podUID="36f0a6a2-54be-49bf-9202-502960cbb134" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded"
Oct 30 22:17:21.363424 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:21.357504    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb0b93/simulator-w6-fb0b93-kserve-55b784b479-588bn" podUID="951a6f7e-a22e-40f2-ace2-1ce5d3bc9b49" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 30 22:17:21.560490 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:21.490753    5678 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-fb0b93/simulator-w6-fb0b93-kserve-55b784b479-588bn" podUID="951a6f7e-a22e-40f2-ace2-1ce5d3bc9b49" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": context deadline exceeded"
Oct 30 22:17:21.560490 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:21.494108    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-65e59f/premium-w4-65e59f-kserve-6dc49468f8-rlzhx" podUID="8590a783-eb24-4759-ad74-1369f36aa9d2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:17:21.560490 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:21.513501    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a00d9b/distinct-2-w1-a00d9b-kserve-68fd95957c-dmgnb" podUID="23a81fda-f7e3-4f91-8d96-9f19c7de6e09" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded"
Oct 30 22:17:21.686701 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:21.686497    5678 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-fb0b93/distinct-2-w6-fb0b93-kserve-7d65bb4698-m6tkh" podUID="6fd276ef-522b-4bb4-b101-0a9c3fe9ecd3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:17:21.686701 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:21.686604    5678 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-jsqpx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:17:21.686701 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:21.686641    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-jsqpx" podUID="bfbe138a-ae9f-45e4-9b30-1bf33a80929c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:17:21.830488 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:21.822933    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb0b93/distinct-2-w6-fb0b93-kserve-7d65bb4698-m6tkh" podUID="6fd276ef-522b-4bb4-b101-0a9c3fe9ecd3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:17:21.902112 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:21.897785    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-65e59f/simulator-w4-65e59f-kserve-85698bcd47-przch" podUID="ee5e843b-17be-4623-8c95-dc07c8b7f506" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:17:21.906758 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:21.906677    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-65e59f/simulator-w4-65e59f-kserve-85698bcd47-przch"
Oct 30 22:17:22.031516 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:22.031486    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-65e59f/premium-w4-65e59f-kserve-6dc49468f8-rlzhx"
Oct 30 22:17:22.078461 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:22.074783    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-fb0b93/simulator-w6-fb0b93-kserve-55b784b479-588bn"
Oct 30 22:17:22.081742 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:22.081547    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a00d9b/distinct-2-w1-a00d9b-kserve-68fd95957c-dmgnb"
Oct 30 22:17:22.121119 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:22.104694    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-65e59f/simulator-w4-65e59f-kserve-85698bcd47-przch"
Oct 30 22:17:22.121119 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:22.114661    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-fb0b93/distinct-2-w6-fb0b93-kserve-7d65bb4698-m6tkh"
Oct 30 22:17:22.121119 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:22.114747    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-7bb1c2/distinct-w3-7bb1c2-kserve-8d695596c-8zgtw"
Oct 30 22:17:22.187025 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:22.186993    5678 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-xq4tp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body=
Oct 30 22:17:22.187184 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:22.187051    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xq4tp" podUID="b2fc93b9-e77d-4306-acec-85ea06f5c4be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused"
Oct 30 22:17:23.187556 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:23.187137    5678 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-xq4tp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body=
Oct 30 22:17:23.187556 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:23.187196    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xq4tp" podUID="b2fc93b9-e77d-4306-acec-85ea06f5c4be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused"
Oct 30 22:17:24.052730 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:24.052698    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e62b0e13-td5c4"]
Oct 30 22:17:24.057498 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:24.057468    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e62b0e13-td5c4"]
Oct 30 22:17:24.543982 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:24.543942    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69be4be8-fc7a-452d-859c-3740d226f614" path="/var/lib/kubelet/pods/69be4be8-fc7a-452d-859c-3740d226f614/volumes"
Oct 30 22:17:24.767488 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:24.767453    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xq4tp"
Oct 30 22:17:24.768033 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:24.768006    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xq4tp"
Oct 30 22:17:24.770483 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:24.770461    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xq4tp"
Oct 30 22:17:35.053000 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:35.052908    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b4f5ba7-4drxr"]
Oct 30 22:17:35.098452 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:35.098422    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b4f5ba7-4drxr"]
Oct 30 22:17:36.526252 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:36.526220    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c68c37eb-ff02-4b46-ac99-6e8b081e091b" path="/var/lib/kubelet/pods/c68c37eb-ff02-4b46-ac99-6e8b081e091b/volumes"
Oct 30 22:17:40.573983 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:17:40.573938    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 22:17:41.059172 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:41.059141    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-67184f2a-59c88bbd6c-j769f"]
Oct 30 22:17:41.059468 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:41.059446    5678 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-67184f2a-59c88bbd6c-j769f" podUID="3129fc82-b397-4f74-a39c-f45e123c4ad1" containerName="maas-api" containerID="cri-o://0515a778d1710d751383c737c4c9975938989992058253db9c2c7bbfbf2ebad9" gracePeriod=30
Oct 30 22:17:41.254288 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:41.254251    5678 generic.go:356] "Generic (PLEG): container finished" podID="3129fc82-b397-4f74-a39c-f45e123c4ad1" containerID="0515a778d1710d751383c737c4c9975938989992058253db9c2c7bbfbf2ebad9" exitCode=0
Oct 30 22:17:41.254464 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:41.254329    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-67184f2a-59c88bbd6c-j769f" event={"ID":"3129fc82-b397-4f74-a39c-f45e123c4ad1","Type":"ContainerDied","Data":"0515a778d1710d751383c737c4c9975938989992058253db9c2c7bbfbf2ebad9"}
Oct 30 22:17:41.418930 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:41.418892    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-67184f2a-59c88bbd6c-j769f"
Oct 30 22:17:41.538754 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:41.538710    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3129fc82-b397-4f74-a39c-f45e123c4ad1-kube-api-access-fsrjh\" (UniqueName: \"kubernetes.io/projected/3129fc82-b397-4f74-a39c-f45e123c4ad1-kube-api-access-fsrjh\") pod \"3129fc82-b397-4f74-a39c-f45e123c4ad1\" (UID: \"3129fc82-b397-4f74-a39c-f45e123c4ad1\") "
Oct 30 22:17:41.539370 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:41.539328    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3129fc82-b397-4f74-a39c-f45e123c4ad1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3129fc82-b397-4f74-a39c-f45e123c4ad1-maas-api-tls\") pod \"3129fc82-b397-4f74-a39c-f45e123c4ad1\" (UID: \"3129fc82-b397-4f74-a39c-f45e123c4ad1\") "
Oct 30 22:17:41.539557 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:41.539389    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3129fc82-b397-4f74-a39c-f45e123c4ad1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3129fc82-b397-4f74-a39c-f45e123c4ad1-metrics-certs\") pod \"3129fc82-b397-4f74-a39c-f45e123c4ad1\" (UID: \"3129fc82-b397-4f74-a39c-f45e123c4ad1\") "
Oct 30 22:17:41.539557 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:41.539527    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3129fc82-b397-4f74-a39c-f45e123c4ad1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3129fc82-b397-4f74-a39c-f45e123c4ad1-openshift-service-ca\") pod \"3129fc82-b397-4f74-a39c-f45e123c4ad1\" (UID: \"3129fc82-b397-4f74-a39c-f45e123c4ad1\") "
Oct 30 22:17:41.542631 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:41.542599    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3129fc82-b397-4f74-a39c-f45e123c4ad1-openshift-service-ca" pod "3129fc82-b397-4f74-a39c-f45e123c4ad1" (UID: "3129fc82-b397-4f74-a39c-f45e123c4ad1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:17:41.547868 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:41.547824    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3129fc82-b397-4f74-a39c-f45e123c4ad1-kube-api-access-fsrjh" pod "3129fc82-b397-4f74-a39c-f45e123c4ad1" (UID: "3129fc82-b397-4f74-a39c-f45e123c4ad1"). InnerVolumeSpecName "kube-api-access-fsrjh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:17:41.558467 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:41.553151    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3129fc82-b397-4f74-a39c-f45e123c4ad1-maas-api-tls" pod "3129fc82-b397-4f74-a39c-f45e123c4ad1" (UID: "3129fc82-b397-4f74-a39c-f45e123c4ad1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:17:41.558467 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:41.557694    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3129fc82-b397-4f74-a39c-f45e123c4ad1-metrics-certs" pod "3129fc82-b397-4f74-a39c-f45e123c4ad1" (UID: "3129fc82-b397-4f74-a39c-f45e123c4ad1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:17:41.642656 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:41.642560    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fsrjh\" (UniqueName: \"kubernetes.io/projected/3129fc82-b397-4f74-a39c-f45e123c4ad1-kube-api-access-fsrjh\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:41.642656 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:41.642605    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3129fc82-b397-4f74-a39c-f45e123c4ad1-maas-api-tls\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:41.642656 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:41.642621    5678 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3129fc82-b397-4f74-a39c-f45e123c4ad1-metrics-certs\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:41.642656 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:41.642635    5678 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3129fc82-b397-4f74-a39c-f45e123c4ad1-openshift-service-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:42.263151 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:42.263123    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-67184f2a-59c88bbd6c-j769f"
Oct 30 22:17:42.263316 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:42.263157    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-67184f2a-59c88bbd6c-j769f" event={"ID":"3129fc82-b397-4f74-a39c-f45e123c4ad1","Type":"ContainerDied","Data":"6810aa0c8d05d09dabc480c192a42d038116c7f7714d1a377792f91628273fe6"}
Oct 30 22:17:42.263316 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:42.263215    5678 scope.go:122] "RemoveContainer" containerID="0515a778d1710d751383c737c4c9975938989992058253db9c2c7bbfbf2ebad9"
Oct 30 22:17:42.423586 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:42.423554    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-67184f2a-59c88bbd6c-j769f"]
Oct 30 22:17:42.471124 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:42.471090    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-67184f2a-59c88bbd6c-j769f"]
Oct 30 22:17:42.525273 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:42.525201    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3129fc82-b397-4f74-a39c-f45e123c4ad1" path="/var/lib/kubelet/pods/3129fc82-b397-4f74-a39c-f45e123c4ad1/volumes"
Oct 30 22:17:45.082695 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:45.082620    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-ac5024ad-openshift-default-65855d67d5-hpztt"]
Oct 30 22:17:45.083692 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:45.083672    5678 cpu_manager.go:414] "containerMap: removing container" podUID="3129fc82-b397-4f74-a39c-f45e123c4ad1" containerName="maas-api"
Oct 30 22:17:45.083753 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:45.083695    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3129fc82-b397-4f74-a39c-f45e123c4ad1" containerName="maas-api"
Oct 30 22:17:45.083951 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:45.083934    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="3129fc82-b397-4f74-a39c-f45e123c4ad1" containerName="maas-api"
Oct 30 22:17:45.088445 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:45.088396    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ac5024ad-openshift-default-65855d67d5-hpztt"
Oct 30 22:17:45.144570 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:45.144547    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-ac5024ad-openshift-default-dockercfg-b69kp\""
Oct 30 22:17:45.197275 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:45.197241    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-ac5024ad-openshift-default-65855d67d5-hpztt"]
Oct 30 22:17:45.200044 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:45.200009    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/669af24d-752d-407e-af9b-8ba5c02c40de-istio-podinfo\") pod \"e2e-ait-ac5024ad-openshift-default-65855d67d5-hpztt\" (UID: \"669af24d-752d-407e-af9b-8ba5c02c40de\") " pod="openshift-ingress/e2e-ait-ac5024ad-openshift-default-65855d67d5-hpztt"
Oct 30 22:17:45.200191 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:45.200149    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/669af24d-752d-407e-af9b-8ba5c02c40de-workload-socket\") pod \"e2e-ait-ac5024ad-openshift-default-65855d67d5-hpztt\" (UID: \"669af24d-752d-407e-af9b-8ba5c02c40de\") " pod="openshift-ingress/e2e-ait-ac5024ad-openshift-default-65855d67d5-hpztt"
Oct 30 22:17:45.200321 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:45.200238    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/669af24d-752d-407e-af9b-8ba5c02c40de-workload-certs\") pod \"e2e-ait-ac5024ad-openshift-default-65855d67d5-hpztt\" (UID: \"669af24d-752d-407e-af9b-8ba5c02c40de\") " pod="openshift-ingress/e2e-ait-ac5024ad-openshift-default-65855d67d5-hpztt"
Oct 30 22:17:45.200392 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:45.200328    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/669af24d-752d-407e-af9b-8ba5c02c40de-istio-data\") pod \"e2e-ait-ac5024ad-openshift-default-65855d67d5-hpztt\" (UID: \"669af24d-752d-407e-af9b-8ba5c02c40de\") " pod="openshift-ingress/e2e-ait-ac5024ad-openshift-default-65855d67d5-hpztt"
Oct 30 22:17:45.200504 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:45.200393    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/669af24d-752d-407e-af9b-8ba5c02c40de-istio-token\") pod \"e2e-ait-ac5024ad-openshift-default-65855d67d5-hpztt\" (UID: \"669af24d-752d-407e-af9b-8ba5c02c40de\") " pod="openshift-ingress/e2e-ait-ac5024ad-openshift-default-65855d67d5-hpztt"
Oct 30 22:17:45.200620 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:45.200598    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/669af24d-752d-407e-af9b-8ba5c02c40de-istio-envoy\") pod \"e2e-ait-ac5024ad-openshift-default-65855d67d5-hpztt\" (UID: \"669af24d-752d-407e-af9b-8ba5c02c40de\") " pod="openshift-ingress/e2e-ait-ac5024ad-openshift-default-65855d67d5-hpztt"
Oct 30 22:17:45.200715 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:45.200698    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2jkv\" (UniqueName: \"kubernetes.io/projected/669af24d-752d-407e-af9b-8ba5c02c40de-kube-api-access-b2jkv\") pod \"e2e-ait-ac5024ad-openshift-default-65855d67d5-hpztt\" (UID: \"669af24d-752d-407e-af9b-8ba5c02c40de\") " pod="openshift-ingress/e2e-ait-ac5024ad-openshift-default-65855d67d5-hpztt"
Oct 30 22:17:45.200780 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:45.200755    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/669af24d-752d-407e-af9b-8ba5c02c40de-credential-socket\") pod \"e2e-ait-ac5024ad-openshift-default-65855d67d5-hpztt\" (UID: \"669af24d-752d-407e-af9b-8ba5c02c40de\") " pod="openshift-ingress/e2e-ait-ac5024ad-openshift-default-65855d67d5-hpztt"
Oct 30 22:17:45.200837 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:45.200818    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/669af24d-752d-407e-af9b-8ba5c02c40de-istiod-ca-cert\") pod \"e2e-ait-ac5024ad-openshift-default-65855d67d5-hpztt\" (UID: \"669af24d-752d-407e-af9b-8ba5c02c40de\") " pod="openshift-ingress/e2e-ait-ac5024ad-openshift-default-65855d67d5-hpztt"
Oct 30 22:17:45.302675 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:45.302643    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/669af24d-752d-407e-af9b-8ba5c02c40de-istio-envoy\") pod \"e2e-ait-ac5024ad-openshift-default-65855d67d5-hpztt\" (UID: \"669af24d-752d-407e-af9b-8ba5c02c40de\") " pod="openshift-ingress/e2e-ait-ac5024ad-openshift-default-65855d67d5-hpztt"
Oct 30 22:17:45.302881 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:45.302719    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2jkv\" (UniqueName: \"kubernetes.io/projected/669af24d-752d-407e-af9b-8ba5c02c40de-kube-api-access-b2jkv\") pod \"e2e-ait-ac5024ad-openshift-default-65855d67d5-hpztt\" (UID: \"669af24d-752d-407e-af9b-8ba5c02c40de\") " pod="openshift-ingress/e2e-ait-ac5024ad-openshift-default-65855d67d5-hpztt"
Oct 30 22:17:45.302881 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:45.302770    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/669af24d-752d-407e-af9b-8ba5c02c40de-credential-socket\") pod \"e2e-ait-ac5024ad-openshift-default-65855d67d5-hpztt\" (UID: \"669af24d-752d-407e-af9b-8ba5c02c40de\") " pod="openshift-ingress/e2e-ait-ac5024ad-openshift-default-65855d67d5-hpztt"
Oct 30 22:17:45.302881 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:45.302809    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/669af24d-752d-407e-af9b-8ba5c02c40de-istiod-ca-cert\") pod \"e2e-ait-ac5024ad-openshift-default-65855d67d5-hpztt\" (UID: \"669af24d-752d-407e-af9b-8ba5c02c40de\") " pod="openshift-ingress/e2e-ait-ac5024ad-openshift-default-65855d67d5-hpztt"
Oct 30 22:17:45.303064 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:45.302898    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/669af24d-752d-407e-af9b-8ba5c02c40de-istio-podinfo\") pod \"e2e-ait-ac5024ad-openshift-default-65855d67d5-hpztt\" (UID: \"669af24d-752d-407e-af9b-8ba5c02c40de\") " pod="openshift-ingress/e2e-ait-ac5024ad-openshift-default-65855d67d5-hpztt"
Oct 30 22:17:45.303064 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:45.302959    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/669af24d-752d-407e-af9b-8ba5c02c40de-workload-socket\") pod \"e2e-ait-ac5024ad-openshift-default-65855d67d5-hpztt\" (UID: \"669af24d-752d-407e-af9b-8ba5c02c40de\") " pod="openshift-ingress/e2e-ait-ac5024ad-openshift-default-65855d67d5-hpztt"
Oct 30 22:17:45.303064 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:45.303016    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/669af24d-752d-407e-af9b-8ba5c02c40de-workload-certs\") pod \"e2e-ait-ac5024ad-openshift-default-65855d67d5-hpztt\" (UID: \"669af24d-752d-407e-af9b-8ba5c02c40de\") " pod="openshift-ingress/e2e-ait-ac5024ad-openshift-default-65855d67d5-hpztt"
Oct 30 22:17:45.303064 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:45.303052    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/669af24d-752d-407e-af9b-8ba5c02c40de-istio-data\") pod \"e2e-ait-ac5024ad-openshift-default-65855d67d5-hpztt\" (UID: \"669af24d-752d-407e-af9b-8ba5c02c40de\") " pod="openshift-ingress/e2e-ait-ac5024ad-openshift-default-65855d67d5-hpztt"
Oct 30 22:17:45.303340 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:45.303123    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/669af24d-752d-407e-af9b-8ba5c02c40de-istio-token\") pod \"e2e-ait-ac5024ad-openshift-default-65855d67d5-hpztt\" (UID: \"669af24d-752d-407e-af9b-8ba5c02c40de\") " pod="openshift-ingress/e2e-ait-ac5024ad-openshift-default-65855d67d5-hpztt"
Oct 30 22:17:45.303340 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:45.303203    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/669af24d-752d-407e-af9b-8ba5c02c40de-credential-socket\") pod \"e2e-ait-ac5024ad-openshift-default-65855d67d5-hpztt\" (UID: \"669af24d-752d-407e-af9b-8ba5c02c40de\") " pod="openshift-ingress/e2e-ait-ac5024ad-openshift-default-65855d67d5-hpztt"
Oct 30 22:17:45.303555 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:45.303529    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/669af24d-752d-407e-af9b-8ba5c02c40de-workload-certs\") pod \"e2e-ait-ac5024ad-openshift-default-65855d67d5-hpztt\" (UID: \"669af24d-752d-407e-af9b-8ba5c02c40de\") " pod="openshift-ingress/e2e-ait-ac5024ad-openshift-default-65855d67d5-hpztt"
Oct 30 22:17:45.303629 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:45.303551    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/669af24d-752d-407e-af9b-8ba5c02c40de-istio-data\") pod \"e2e-ait-ac5024ad-openshift-default-65855d67d5-hpztt\" (UID: \"669af24d-752d-407e-af9b-8ba5c02c40de\") " pod="openshift-ingress/e2e-ait-ac5024ad-openshift-default-65855d67d5-hpztt"
Oct 30 22:17:45.303929 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:45.303909    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/669af24d-752d-407e-af9b-8ba5c02c40de-istiod-ca-cert\") pod \"e2e-ait-ac5024ad-openshift-default-65855d67d5-hpztt\" (UID: \"669af24d-752d-407e-af9b-8ba5c02c40de\") " pod="openshift-ingress/e2e-ait-ac5024ad-openshift-default-65855d67d5-hpztt"
Oct 30 22:17:45.304079 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:45.304063    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/669af24d-752d-407e-af9b-8ba5c02c40de-workload-socket\") pod \"e2e-ait-ac5024ad-openshift-default-65855d67d5-hpztt\" (UID: \"669af24d-752d-407e-af9b-8ba5c02c40de\") " pod="openshift-ingress/e2e-ait-ac5024ad-openshift-default-65855d67d5-hpztt"
Oct 30 22:17:45.305117 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:45.305089    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/669af24d-752d-407e-af9b-8ba5c02c40de-istio-envoy\") pod \"e2e-ait-ac5024ad-openshift-default-65855d67d5-hpztt\" (UID: \"669af24d-752d-407e-af9b-8ba5c02c40de\") " pod="openshift-ingress/e2e-ait-ac5024ad-openshift-default-65855d67d5-hpztt"
Oct 30 22:17:45.305595 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:45.305576    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/669af24d-752d-407e-af9b-8ba5c02c40de-istio-podinfo\") pod \"e2e-ait-ac5024ad-openshift-default-65855d67d5-hpztt\" (UID: \"669af24d-752d-407e-af9b-8ba5c02c40de\") " pod="openshift-ingress/e2e-ait-ac5024ad-openshift-default-65855d67d5-hpztt"
Oct 30 22:17:45.362306 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:45.362245    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/669af24d-752d-407e-af9b-8ba5c02c40de-istio-token\") pod \"e2e-ait-ac5024ad-openshift-default-65855d67d5-hpztt\" (UID: \"669af24d-752d-407e-af9b-8ba5c02c40de\") " pod="openshift-ingress/e2e-ait-ac5024ad-openshift-default-65855d67d5-hpztt"
Oct 30 22:17:45.362438 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:45.362417    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2jkv\" (UniqueName: \"kubernetes.io/projected/669af24d-752d-407e-af9b-8ba5c02c40de-kube-api-access-b2jkv\") pod \"e2e-ait-ac5024ad-openshift-default-65855d67d5-hpztt\" (UID: \"669af24d-752d-407e-af9b-8ba5c02c40de\") " pod="openshift-ingress/e2e-ait-ac5024ad-openshift-default-65855d67d5-hpztt"
Oct 30 22:17:45.412896 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:45.412010    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ac5024ad-openshift-default-65855d67d5-hpztt"
Oct 30 22:17:46.065990 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:46.065893    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-ac5024ad-openshift-default-65855d67d5-hpztt"]
Oct 30 22:17:46.071017 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:17:46.070911    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod669af24d_752d_407e_af9b_8ba5c02c40de.slice/crio-8e58a9f2f952413640658623bd116f8b10951368f20c9521f0876e68373eaaa1 WatchSource:0}: Error finding container 8e58a9f2f952413640658623bd116f8b10951368f20c9521f0876e68373eaaa1: Status 404 returned error can't find the container with id 8e58a9f2f952413640658623bd116f8b10951368f20c9521f0876e68373eaaa1
Oct 30 22:17:46.074174 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:46.074131    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:17:46.074290 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:46.074225    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:17:46.074290 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:46.074270    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:17:46.307658 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:46.307620    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ac5024ad-openshift-default-65855d67d5-hpztt" event={"ID":"669af24d-752d-407e-af9b-8ba5c02c40de","Type":"ContainerStarted","Data":"7ee0cb93980198060bf70153267f3892dc5b97235ca62c721304ba81388cf339"}
Oct 30 22:17:46.308396 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:46.307684    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ac5024ad-openshift-default-65855d67d5-hpztt" event={"ID":"669af24d-752d-407e-af9b-8ba5c02c40de","Type":"ContainerStarted","Data":"8e58a9f2f952413640658623bd116f8b10951368f20c9521f0876e68373eaaa1"}
Oct 30 22:17:46.413366 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:46.413336    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-ac5024ad-openshift-default-65855d67d5-hpztt"
Oct 30 22:17:46.429286 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:46.429219    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-ac5024ad-openshift-default-65855d67d5-hpztt" podStartSLOduration=2.42920471 podStartE2EDuration="2.42920471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:17:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:17:46.425171153 +0000 UTC m=+1644.029478728" watchObservedRunningTime="2026-10-30 22:17:46.42920471 +0000 UTC m=+1644.033512263"
Oct 30 22:17:47.418300 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:47.418268    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-ac5024ad-openshift-default-65855d67d5-hpztt"
Oct 30 22:17:48.324302 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:48.324273    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-ac5024ad-openshift-default-65855d67d5-hpztt"
Oct 30 22:17:48.325160 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:48.325137    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-ac5024ad-openshift-default-65855d67d5-hpztt"
Oct 30 22:17:48.948188 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:48.948058    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-72b5472b-6557fcb676-2x9c8"]
Oct 30 22:17:48.948715 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:48.948308    5678 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-72b5472b-6557fcb676-2x9c8" podUID="5f6b333b-fdc7-4327-88e7-43272ca7cc51" containerName="maas-api" containerID="cri-o://3672ce24469d34b98e2a26086196ac9f07c67598b574197664b157e87ea3d19e" gracePeriod=30
Oct 30 22:17:49.310856 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:49.310823    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-72b5472b-6557fcb676-2x9c8"
Oct 30 22:17:49.335214 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:49.335181    5678 generic.go:356] "Generic (PLEG): container finished" podID="5f6b333b-fdc7-4327-88e7-43272ca7cc51" containerID="3672ce24469d34b98e2a26086196ac9f07c67598b574197664b157e87ea3d19e" exitCode=0
Oct 30 22:17:49.336198 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:49.336173    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-72b5472b-6557fcb676-2x9c8"
Oct 30 22:17:49.336349 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:49.336309    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-72b5472b-6557fcb676-2x9c8" event={"ID":"5f6b333b-fdc7-4327-88e7-43272ca7cc51","Type":"ContainerDied","Data":"3672ce24469d34b98e2a26086196ac9f07c67598b574197664b157e87ea3d19e"}
Oct 30 22:17:49.336472 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:49.336351    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-72b5472b-6557fcb676-2x9c8" event={"ID":"5f6b333b-fdc7-4327-88e7-43272ca7cc51","Type":"ContainerDied","Data":"5a2044ce72d786f357f0b42a54f887184460fe67606df5d41bf3ba464fefb13e"}
Oct 30 22:17:49.336472 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:49.336380    5678 scope.go:122] "RemoveContainer" containerID="3672ce24469d34b98e2a26086196ac9f07c67598b574197664b157e87ea3d19e"
Oct 30 22:17:49.351598 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:49.351580    5678 scope.go:122] "RemoveContainer" containerID="3672ce24469d34b98e2a26086196ac9f07c67598b574197664b157e87ea3d19e"
Oct 30 22:17:49.351933 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:17:49.351913    5678 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3672ce24469d34b98e2a26086196ac9f07c67598b574197664b157e87ea3d19e\": container with ID starting with 3672ce24469d34b98e2a26086196ac9f07c67598b574197664b157e87ea3d19e not found: ID does not exist" containerID="3672ce24469d34b98e2a26086196ac9f07c67598b574197664b157e87ea3d19e"
Oct 30 22:17:49.351995 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:49.351945    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3672ce24469d34b98e2a26086196ac9f07c67598b574197664b157e87ea3d19e"} err="failed to get container status \"3672ce24469d34b98e2a26086196ac9f07c67598b574197664b157e87ea3d19e\": rpc error: code = NotFound desc = could not find container \"3672ce24469d34b98e2a26086196ac9f07c67598b574197664b157e87ea3d19e\": container with ID starting with 3672ce24469d34b98e2a26086196ac9f07c67598b574197664b157e87ea3d19e not found: ID does not exist"
Oct 30 22:17:49.415134 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:49.411095    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f6b333b-fdc7-4327-88e7-43272ca7cc51-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f6b333b-fdc7-4327-88e7-43272ca7cc51-openshift-service-ca\") pod \"5f6b333b-fdc7-4327-88e7-43272ca7cc51\" (UID: \"5f6b333b-fdc7-4327-88e7-43272ca7cc51\") "
Oct 30 22:17:49.415134 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:49.411233    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f6b333b-fdc7-4327-88e7-43272ca7cc51-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f6b333b-fdc7-4327-88e7-43272ca7cc51-metrics-certs\") pod \"5f6b333b-fdc7-4327-88e7-43272ca7cc51\" (UID: \"5f6b333b-fdc7-4327-88e7-43272ca7cc51\") "
Oct 30 22:17:49.415134 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:49.411287    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f6b333b-fdc7-4327-88e7-43272ca7cc51-kube-api-access-cxzdp\" (UniqueName: \"kubernetes.io/projected/5f6b333b-fdc7-4327-88e7-43272ca7cc51-kube-api-access-cxzdp\") pod \"5f6b333b-fdc7-4327-88e7-43272ca7cc51\" (UID: \"5f6b333b-fdc7-4327-88e7-43272ca7cc51\") "
Oct 30 22:17:49.415134 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:49.411336    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f6b333b-fdc7-4327-88e7-43272ca7cc51-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f6b333b-fdc7-4327-88e7-43272ca7cc51-maas-api-tls\") pod \"5f6b333b-fdc7-4327-88e7-43272ca7cc51\" (UID: \"5f6b333b-fdc7-4327-88e7-43272ca7cc51\") "
Oct 30 22:17:49.416317 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:49.415891    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f6b333b-fdc7-4327-88e7-43272ca7cc51-openshift-service-ca" pod "5f6b333b-fdc7-4327-88e7-43272ca7cc51" (UID: "5f6b333b-fdc7-4327-88e7-43272ca7cc51"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:17:49.433554 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:49.422492    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f6b333b-fdc7-4327-88e7-43272ca7cc51-kube-api-access-cxzdp" pod "5f6b333b-fdc7-4327-88e7-43272ca7cc51" (UID: "5f6b333b-fdc7-4327-88e7-43272ca7cc51"). InnerVolumeSpecName "kube-api-access-cxzdp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:17:49.433554 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:49.422503    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f6b333b-fdc7-4327-88e7-43272ca7cc51-maas-api-tls" pod "5f6b333b-fdc7-4327-88e7-43272ca7cc51" (UID: "5f6b333b-fdc7-4327-88e7-43272ca7cc51"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:17:49.433554 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:49.423116    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f6b333b-fdc7-4327-88e7-43272ca7cc51-metrics-certs" pod "5f6b333b-fdc7-4327-88e7-43272ca7cc51" (UID: "5f6b333b-fdc7-4327-88e7-43272ca7cc51"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:17:49.514246 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:49.514216    5678 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f6b333b-fdc7-4327-88e7-43272ca7cc51-openshift-service-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:49.514246 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:49.514247    5678 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f6b333b-fdc7-4327-88e7-43272ca7cc51-metrics-certs\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:49.514246 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:49.514256    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxzdp\" (UniqueName: \"kubernetes.io/projected/5f6b333b-fdc7-4327-88e7-43272ca7cc51-kube-api-access-cxzdp\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:49.514596 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:49.514267    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f6b333b-fdc7-4327-88e7-43272ca7cc51-maas-api-tls\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:49.756612 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:49.756571    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-72b5472b-6557fcb676-2x9c8"]
Oct 30 22:17:49.762526 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:49.762497    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-72b5472b-6557fcb676-2x9c8"]
Oct 30 22:17:50.526989 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:50.526959    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f6b333b-fdc7-4327-88e7-43272ca7cc51" path="/var/lib/kubelet/pods/5f6b333b-fdc7-4327-88e7-43272ca7cc51/volumes"
Oct 30 22:17:50.823344 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:50.823262    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f198c49a-b2nsb"]
Oct 30 22:17:50.825040 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:50.825012    5678 cpu_manager.go:414] "containerMap: removing container" podUID="5f6b333b-fdc7-4327-88e7-43272ca7cc51" containerName="maas-api"
Oct 30 22:17:50.825218 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:50.825204    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f6b333b-fdc7-4327-88e7-43272ca7cc51" containerName="maas-api"
Oct 30 22:17:50.825545 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:50.825527    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f6b333b-fdc7-4327-88e7-43272ca7cc51" containerName="maas-api"
Oct 30 22:17:50.830219 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:50.830199    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f198c49a-b2nsb"
Oct 30 22:17:50.865969 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:50.865942    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-ff716ef6-openshift-default-5bbc788795-f68kp"]
Oct 30 22:17:50.866144 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:50.866125    5678 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-ff716ef6-openshift-default-5bbc788795-f68kp" podUID="313acbe0-dc5c-48cb-92d8-972092227317" containerName="istio-proxy" containerID="cri-o://99163503e6d645c29cca4298bd14c39826e7f00ab72ec6e5efa30a7d73208cf4" gracePeriod=30
Oct 30 22:17:50.887909 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:50.887889    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5rvcj\""
Oct 30 22:17:50.888503 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:50.888487    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:17:50.889101 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:50.889080    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f198c49a-b2nsb"]
Oct 30 22:17:50.957891 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:50.953916    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ppms\" (UniqueName: \"kubernetes.io/projected/aa29d2ac-60f7-4a90-8d2c-4161eccb6a1d-kube-api-access-5ppms\") pod \"maas-api-revoke-sub-f198c49a-b2nsb\" (UID: \"aa29d2ac-60f7-4a90-8d2c-4161eccb6a1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f198c49a-b2nsb"
Oct 30 22:17:50.958232 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:50.958200    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa29d2ac-60f7-4a90-8d2c-4161eccb6a1d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f198c49a-b2nsb\" (UID: \"aa29d2ac-60f7-4a90-8d2c-4161eccb6a1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f198c49a-b2nsb"
Oct 30 22:17:50.962389 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:50.962366    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa29d2ac-60f7-4a90-8d2c-4161eccb6a1d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f198c49a-b2nsb\" (UID: \"aa29d2ac-60f7-4a90-8d2c-4161eccb6a1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f198c49a-b2nsb"
Oct 30 22:17:51.069053 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:51.068508    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ppms\" (UniqueName: \"kubernetes.io/projected/aa29d2ac-60f7-4a90-8d2c-4161eccb6a1d-kube-api-access-5ppms\") pod \"maas-api-revoke-sub-f198c49a-b2nsb\" (UID: \"aa29d2ac-60f7-4a90-8d2c-4161eccb6a1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f198c49a-b2nsb"
Oct 30 22:17:51.069376 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:51.069349    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa29d2ac-60f7-4a90-8d2c-4161eccb6a1d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f198c49a-b2nsb\" (UID: \"aa29d2ac-60f7-4a90-8d2c-4161eccb6a1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f198c49a-b2nsb"
Oct 30 22:17:51.069590 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:51.069572    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa29d2ac-60f7-4a90-8d2c-4161eccb6a1d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f198c49a-b2nsb\" (UID: \"aa29d2ac-60f7-4a90-8d2c-4161eccb6a1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f198c49a-b2nsb"
Oct 30 22:17:51.074083 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:51.073055    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa29d2ac-60f7-4a90-8d2c-4161eccb6a1d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f198c49a-b2nsb\" (UID: \"aa29d2ac-60f7-4a90-8d2c-4161eccb6a1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f198c49a-b2nsb"
Oct 30 22:17:51.074083 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:51.074006    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa29d2ac-60f7-4a90-8d2c-4161eccb6a1d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f198c49a-b2nsb\" (UID: \"aa29d2ac-60f7-4a90-8d2c-4161eccb6a1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f198c49a-b2nsb"
Oct 30 22:17:51.148122 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:51.148079    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ppms\" (UniqueName: \"kubernetes.io/projected/aa29d2ac-60f7-4a90-8d2c-4161eccb6a1d-kube-api-access-5ppms\") pod \"maas-api-revoke-sub-f198c49a-b2nsb\" (UID: \"aa29d2ac-60f7-4a90-8d2c-4161eccb6a1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f198c49a-b2nsb"
Oct 30 22:17:51.444344 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:51.444312    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f198c49a-b2nsb"
Oct 30 22:17:51.921228 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:17:51.921189    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa29d2ac_60f7_4a90_8d2c_4161eccb6a1d.slice/crio-0c4d4767846c05782b8db0fbac98a6bfa8bb1d7b6d23fc762a78f43948fe7201 WatchSource:0}: Error finding container 0c4d4767846c05782b8db0fbac98a6bfa8bb1d7b6d23fc762a78f43948fe7201: Status 404 returned error can't find the container with id 0c4d4767846c05782b8db0fbac98a6bfa8bb1d7b6d23fc762a78f43948fe7201
Oct 30 22:17:51.922539 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:51.922518    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f198c49a-b2nsb"]
Oct 30 22:17:52.367546 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:52.367514    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f198c49a-b2nsb" event={"ID":"aa29d2ac-60f7-4a90-8d2c-4161eccb6a1d","Type":"ContainerStarted","Data":"0c4d4767846c05782b8db0fbac98a6bfa8bb1d7b6d23fc762a78f43948fe7201"}
Oct 30 22:17:53.420599 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:53.420557    5678 generic.go:356] "Generic (PLEG): container finished" podID="aa29d2ac-60f7-4a90-8d2c-4161eccb6a1d" containerID="02209031bd309f6b04da1ea80db58be2eca85299cbb30f72627a675a7a7a10de" exitCode=0
Oct 30 22:17:53.421112 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:53.420728    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f198c49a-b2nsb" event={"ID":"aa29d2ac-60f7-4a90-8d2c-4161eccb6a1d","Type":"ContainerDied","Data":"02209031bd309f6b04da1ea80db58be2eca85299cbb30f72627a675a7a7a10de"}
Oct 30 22:17:54.055719 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:54.055685    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34a1fc25-hmlzl"]
Oct 30 22:17:54.060489 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:54.060461    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34a1fc25-hmlzl"]
Oct 30 22:17:54.664853 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:54.664826    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66b4ab22-b3fa-4ed8-9ade-370752cf2041" path="/var/lib/kubelet/pods/66b4ab22-b3fa-4ed8-9ade-370752cf2041/volumes"
Oct 30 22:17:54.732062 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:54.732035    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f198c49a-b2nsb"
Oct 30 22:17:54.815863 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:54.815829    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa29d2ac-60f7-4a90-8d2c-4161eccb6a1d-kube-api-access-5ppms\" (UniqueName: \"kubernetes.io/projected/aa29d2ac-60f7-4a90-8d2c-4161eccb6a1d-kube-api-access-5ppms\") pod \"aa29d2ac-60f7-4a90-8d2c-4161eccb6a1d\" (UID: \"aa29d2ac-60f7-4a90-8d2c-4161eccb6a1d\") "
Oct 30 22:17:54.816197 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:54.816176    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa29d2ac-60f7-4a90-8d2c-4161eccb6a1d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa29d2ac-60f7-4a90-8d2c-4161eccb6a1d-maas-api-openshift-ca\") pod \"aa29d2ac-60f7-4a90-8d2c-4161eccb6a1d\" (UID: \"aa29d2ac-60f7-4a90-8d2c-4161eccb6a1d\") "
Oct 30 22:17:54.816340 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:54.816323    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa29d2ac-60f7-4a90-8d2c-4161eccb6a1d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa29d2ac-60f7-4a90-8d2c-4161eccb6a1d-maas-api-xks-ca\") pod \"aa29d2ac-60f7-4a90-8d2c-4161eccb6a1d\" (UID: \"aa29d2ac-60f7-4a90-8d2c-4161eccb6a1d\") "
Oct 30 22:17:54.816531 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:54.816504    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa29d2ac-60f7-4a90-8d2c-4161eccb6a1d-maas-api-openshift-ca" pod "aa29d2ac-60f7-4a90-8d2c-4161eccb6a1d" (UID: "aa29d2ac-60f7-4a90-8d2c-4161eccb6a1d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:17:54.817141 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:54.817114    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa29d2ac-60f7-4a90-8d2c-4161eccb6a1d-maas-api-openshift-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:54.818065 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:54.818044    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa29d2ac-60f7-4a90-8d2c-4161eccb6a1d-kube-api-access-5ppms" pod "aa29d2ac-60f7-4a90-8d2c-4161eccb6a1d" (UID: "aa29d2ac-60f7-4a90-8d2c-4161eccb6a1d"). InnerVolumeSpecName "kube-api-access-5ppms". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:17:54.818568 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:54.818545    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa29d2ac-60f7-4a90-8d2c-4161eccb6a1d-maas-api-xks-ca" pod "aa29d2ac-60f7-4a90-8d2c-4161eccb6a1d" (UID: "aa29d2ac-60f7-4a90-8d2c-4161eccb6a1d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:17:54.921030 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:54.920924    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa29d2ac-60f7-4a90-8d2c-4161eccb6a1d-maas-api-xks-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:54.921030 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:54.920959    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ppms\" (UniqueName: \"kubernetes.io/projected/aa29d2ac-60f7-4a90-8d2c-4161eccb6a1d-kube-api-access-5ppms\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:55.440786 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:55.440748    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f198c49a-b2nsb"
Oct 30 22:17:55.440786 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:55.440758    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f198c49a-b2nsb" event={"ID":"aa29d2ac-60f7-4a90-8d2c-4161eccb6a1d","Type":"ContainerDied","Data":"0c4d4767846c05782b8db0fbac98a6bfa8bb1d7b6d23fc762a78f43948fe7201"}
Oct 30 22:17:55.440985 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:55.440835    5678 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0c4d4767846c05782b8db0fbac98a6bfa8bb1d7b6d23fc762a78f43948fe7201"
Oct 30 22:17:56.146660 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:56.146606    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ff716ef6-openshift-default-5bbc788795-f68kp"
Oct 30 22:17:56.240350 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:56.240319    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/313acbe0-dc5c-48cb-92d8-972092227317-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/313acbe0-dc5c-48cb-92d8-972092227317-istio-podinfo\") pod \"313acbe0-dc5c-48cb-92d8-972092227317\" (UID: \"313acbe0-dc5c-48cb-92d8-972092227317\") "
Oct 30 22:17:56.240545 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:56.240362    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/313acbe0-dc5c-48cb-92d8-972092227317-istio-token\" (UniqueName: \"kubernetes.io/projected/313acbe0-dc5c-48cb-92d8-972092227317-istio-token\") pod \"313acbe0-dc5c-48cb-92d8-972092227317\" (UID: \"313acbe0-dc5c-48cb-92d8-972092227317\") "
Oct 30 22:17:56.240545 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:56.240485    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/313acbe0-dc5c-48cb-92d8-972092227317-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/313acbe0-dc5c-48cb-92d8-972092227317-credential-socket\") pod \"313acbe0-dc5c-48cb-92d8-972092227317\" (UID: \"313acbe0-dc5c-48cb-92d8-972092227317\") "
Oct 30 22:17:56.240545 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:56.240505    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/313acbe0-dc5c-48cb-92d8-972092227317-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/313acbe0-dc5c-48cb-92d8-972092227317-istiod-ca-cert\") pod \"313acbe0-dc5c-48cb-92d8-972092227317\" (UID: \"313acbe0-dc5c-48cb-92d8-972092227317\") "
Oct 30 22:17:56.240545 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:56.240538    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/313acbe0-dc5c-48cb-92d8-972092227317-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/313acbe0-dc5c-48cb-92d8-972092227317-istio-envoy\") pod \"313acbe0-dc5c-48cb-92d8-972092227317\" (UID: \"313acbe0-dc5c-48cb-92d8-972092227317\") "
Oct 30 22:17:56.240769 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:56.240562    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/313acbe0-dc5c-48cb-92d8-972092227317-kube-api-access-f42nb\" (UniqueName: \"kubernetes.io/projected/313acbe0-dc5c-48cb-92d8-972092227317-kube-api-access-f42nb\") pod \"313acbe0-dc5c-48cb-92d8-972092227317\" (UID: \"313acbe0-dc5c-48cb-92d8-972092227317\") "
Oct 30 22:17:56.240769 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:56.240673    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/313acbe0-dc5c-48cb-92d8-972092227317-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/313acbe0-dc5c-48cb-92d8-972092227317-workload-socket\") pod \"313acbe0-dc5c-48cb-92d8-972092227317\" (UID: \"313acbe0-dc5c-48cb-92d8-972092227317\") "
Oct 30 22:17:56.240769 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:56.240737    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/313acbe0-dc5c-48cb-92d8-972092227317-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/313acbe0-dc5c-48cb-92d8-972092227317-workload-certs\") pod \"313acbe0-dc5c-48cb-92d8-972092227317\" (UID: \"313acbe0-dc5c-48cb-92d8-972092227317\") "
Oct 30 22:17:56.240928 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:56.240810    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/313acbe0-dc5c-48cb-92d8-972092227317-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/313acbe0-dc5c-48cb-92d8-972092227317-istio-data\") pod \"313acbe0-dc5c-48cb-92d8-972092227317\" (UID: \"313acbe0-dc5c-48cb-92d8-972092227317\") "
Oct 30 22:17:56.240928 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:56.240816    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/313acbe0-dc5c-48cb-92d8-972092227317-credential-socket" pod "313acbe0-dc5c-48cb-92d8-972092227317" (UID: "313acbe0-dc5c-48cb-92d8-972092227317"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:17:56.241052 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:56.240944    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/313acbe0-dc5c-48cb-92d8-972092227317-istiod-ca-cert" pod "313acbe0-dc5c-48cb-92d8-972092227317" (UID: "313acbe0-dc5c-48cb-92d8-972092227317"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:17:56.241112 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:56.241029    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/313acbe0-dc5c-48cb-92d8-972092227317-workload-socket" pod "313acbe0-dc5c-48cb-92d8-972092227317" (UID: "313acbe0-dc5c-48cb-92d8-972092227317"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:17:56.241215 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:56.241190    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/313acbe0-dc5c-48cb-92d8-972092227317-workload-certs" pod "313acbe0-dc5c-48cb-92d8-972092227317" (UID: "313acbe0-dc5c-48cb-92d8-972092227317"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:17:56.241316 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:56.241296    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/313acbe0-dc5c-48cb-92d8-972092227317-istio-data" pod "313acbe0-dc5c-48cb-92d8-972092227317" (UID: "313acbe0-dc5c-48cb-92d8-972092227317"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:17:56.241802 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:56.241774    5678 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/313acbe0-dc5c-48cb-92d8-972092227317-credential-socket\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:56.241941 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:56.241804    5678 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/313acbe0-dc5c-48cb-92d8-972092227317-istiod-ca-cert\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:56.241941 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:56.241819    5678 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/313acbe0-dc5c-48cb-92d8-972092227317-workload-socket\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:56.241941 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:56.241836    5678 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/313acbe0-dc5c-48cb-92d8-972092227317-workload-certs\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:56.241941 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:56.241849    5678 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/313acbe0-dc5c-48cb-92d8-972092227317-istio-data\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:56.242674 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:56.242650    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/313acbe0-dc5c-48cb-92d8-972092227317-istio-token" pod "313acbe0-dc5c-48cb-92d8-972092227317" (UID: "313acbe0-dc5c-48cb-92d8-972092227317"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:17:56.242798 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:56.242693    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/313acbe0-dc5c-48cb-92d8-972092227317-istio-podinfo" pod "313acbe0-dc5c-48cb-92d8-972092227317" (UID: "313acbe0-dc5c-48cb-92d8-972092227317"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:17:56.242866 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:56.242799    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/313acbe0-dc5c-48cb-92d8-972092227317-istio-envoy" pod "313acbe0-dc5c-48cb-92d8-972092227317" (UID: "313acbe0-dc5c-48cb-92d8-972092227317"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:17:56.243112 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:56.243095    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/313acbe0-dc5c-48cb-92d8-972092227317-kube-api-access-f42nb" pod "313acbe0-dc5c-48cb-92d8-972092227317" (UID: "313acbe0-dc5c-48cb-92d8-972092227317"). InnerVolumeSpecName "kube-api-access-f42nb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:17:56.344051 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:56.344008    5678 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/313acbe0-dc5c-48cb-92d8-972092227317-istio-podinfo\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:56.344051 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:56.344047    5678 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/313acbe0-dc5c-48cb-92d8-972092227317-istio-token\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:56.344051 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:56.344057    5678 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/313acbe0-dc5c-48cb-92d8-972092227317-istio-envoy\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:56.344365 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:56.344066    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f42nb\" (UniqueName: \"kubernetes.io/projected/313acbe0-dc5c-48cb-92d8-972092227317-kube-api-access-f42nb\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:56.468131 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:56.468041    5678 generic.go:356] "Generic (PLEG): container finished" podID="313acbe0-dc5c-48cb-92d8-972092227317" containerID="99163503e6d645c29cca4298bd14c39826e7f00ab72ec6e5efa30a7d73208cf4" exitCode=0
Oct 30 22:17:56.468131 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:56.468089    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ff716ef6-openshift-default-5bbc788795-f68kp"
Oct 30 22:17:56.468131 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:56.468114    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ff716ef6-openshift-default-5bbc788795-f68kp" event={"ID":"313acbe0-dc5c-48cb-92d8-972092227317","Type":"ContainerDied","Data":"99163503e6d645c29cca4298bd14c39826e7f00ab72ec6e5efa30a7d73208cf4"}
Oct 30 22:17:56.468388 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:56.468181    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ff716ef6-openshift-default-5bbc788795-f68kp" event={"ID":"313acbe0-dc5c-48cb-92d8-972092227317","Type":"ContainerDied","Data":"434d9558dec72c398263b19b0f4e6fbd7e4f8ef43ebd249636a035f804c50a6f"}
Oct 30 22:17:56.468388 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:56.468202    5678 scope.go:122] "RemoveContainer" containerID="99163503e6d645c29cca4298bd14c39826e7f00ab72ec6e5efa30a7d73208cf4"
Oct 30 22:17:56.482854 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:56.482821    5678 scope.go:122] "RemoveContainer" containerID="141a472a7e05afdc52b72cd370f0ad46fc2206ebc9a7bae42c951e83738cc0fb"
Oct 30 22:17:56.497120 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:56.497081    5678 scope.go:122] "RemoveContainer" containerID="99163503e6d645c29cca4298bd14c39826e7f00ab72ec6e5efa30a7d73208cf4"
Oct 30 22:17:56.497526 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:17:56.497493    5678 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99163503e6d645c29cca4298bd14c39826e7f00ab72ec6e5efa30a7d73208cf4\": container with ID starting with 99163503e6d645c29cca4298bd14c39826e7f00ab72ec6e5efa30a7d73208cf4 not found: ID does not exist" containerID="99163503e6d645c29cca4298bd14c39826e7f00ab72ec6e5efa30a7d73208cf4"
Oct 30 22:17:56.497645 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:56.497536    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99163503e6d645c29cca4298bd14c39826e7f00ab72ec6e5efa30a7d73208cf4"} err="failed to get container status \"99163503e6d645c29cca4298bd14c39826e7f00ab72ec6e5efa30a7d73208cf4\": rpc error: code = NotFound desc = could not find container \"99163503e6d645c29cca4298bd14c39826e7f00ab72ec6e5efa30a7d73208cf4\": container with ID starting with 99163503e6d645c29cca4298bd14c39826e7f00ab72ec6e5efa30a7d73208cf4 not found: ID does not exist"
Oct 30 22:17:56.497645 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:56.497559    5678 scope.go:122] "RemoveContainer" containerID="141a472a7e05afdc52b72cd370f0ad46fc2206ebc9a7bae42c951e83738cc0fb"
Oct 30 22:17:56.497953 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:17:56.497905    5678 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"141a472a7e05afdc52b72cd370f0ad46fc2206ebc9a7bae42c951e83738cc0fb\": container with ID starting with 141a472a7e05afdc52b72cd370f0ad46fc2206ebc9a7bae42c951e83738cc0fb not found: ID does not exist" containerID="141a472a7e05afdc52b72cd370f0ad46fc2206ebc9a7bae42c951e83738cc0fb"
Oct 30 22:17:56.497953 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:56.497937    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"141a472a7e05afdc52b72cd370f0ad46fc2206ebc9a7bae42c951e83738cc0fb"} err="failed to get container status \"141a472a7e05afdc52b72cd370f0ad46fc2206ebc9a7bae42c951e83738cc0fb\": rpc error: code = NotFound desc = could not find container \"141a472a7e05afdc52b72cd370f0ad46fc2206ebc9a7bae42c951e83738cc0fb\": container with ID starting with 141a472a7e05afdc52b72cd370f0ad46fc2206ebc9a7bae42c951e83738cc0fb not found: ID does not exist"
Oct 30 22:17:56.646091 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:56.646059    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-ff716ef6-openshift-default-5bbc788795-f68kp"]
Oct 30 22:17:56.652472 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:56.652375    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-ff716ef6-openshift-default-5bbc788795-f68kp"]
Oct 30 22:17:57.300754 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:57.300719    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-c42c0a-55d8fcbd6c-rg6mj"]
Oct 30 22:17:57.301381 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:57.301364    5678 cpu_manager.go:414] "containerMap: removing container" podUID="aa29d2ac-60f7-4a90-8d2c-4161eccb6a1d" containerName="revoke-keys"
Oct 30 22:17:57.301381 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:57.301381    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa29d2ac-60f7-4a90-8d2c-4161eccb6a1d" containerName="revoke-keys"
Oct 30 22:17:57.301528 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:57.301417    5678 cpu_manager.go:414] "containerMap: removing container" podUID="313acbe0-dc5c-48cb-92d8-972092227317" containerName="istio-proxy"
Oct 30 22:17:57.301528 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:57.301426    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="313acbe0-dc5c-48cb-92d8-972092227317" containerName="istio-proxy"
Oct 30 22:17:57.301528 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:57.301438    5678 cpu_manager.go:414] "containerMap: removing container" podUID="313acbe0-dc5c-48cb-92d8-972092227317" containerName="istio-proxy"
Oct 30 22:17:57.301528 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:57.301446    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="313acbe0-dc5c-48cb-92d8-972092227317" containerName="istio-proxy"
Oct 30 22:17:57.301648 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:57.301597    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="313acbe0-dc5c-48cb-92d8-972092227317" containerName="istio-proxy"
Oct 30 22:17:57.301648 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:57.301608    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="313acbe0-dc5c-48cb-92d8-972092227317" containerName="istio-proxy"
Oct 30 22:17:57.301648 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:57.301624    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa29d2ac-60f7-4a90-8d2c-4161eccb6a1d" containerName="revoke-keys"
Oct 30 22:17:57.304927 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:57.304906    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-c42c0a-55d8fcbd6c-rg6mj"
Oct 30 22:17:57.378424 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:57.376803    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-c42c0a-55d8fcbd6c-rg6mj"]
Oct 30 22:17:57.387191 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:57.387161    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-c42c0a\""
Oct 30 22:17:57.387572 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:57.387548    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-c42c0a-dockercfg-d579c\""
Oct 30 22:17:57.391989 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:57.391960    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf53af2e-175e-4af2-a5a4-144853f6af11-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-c42c0a-55d8fcbd6c-rg6mj\" (UID: \"bf53af2e-175e-4af2-a5a4-144853f6af11\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-c42c0a-55d8fcbd6c-rg6mj"
Oct 30 22:17:57.392261 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:57.392233    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zm5b\" (UniqueName: \"kubernetes.io/projected/bf53af2e-175e-4af2-a5a4-144853f6af11-kube-api-access-8zm5b\") pod \"payload-processing-e2e-shared-a-w0-c42c0a-55d8fcbd6c-rg6mj\" (UID: \"bf53af2e-175e-4af2-a5a4-144853f6af11\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-c42c0a-55d8fcbd6c-rg6mj"
Oct 30 22:17:57.516300 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:57.516257    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf53af2e-175e-4af2-a5a4-144853f6af11-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-c42c0a-55d8fcbd6c-rg6mj\" (UID: \"bf53af2e-175e-4af2-a5a4-144853f6af11\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-c42c0a-55d8fcbd6c-rg6mj"
Oct 30 22:17:57.516546 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:57.516379    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zm5b\" (UniqueName: \"kubernetes.io/projected/bf53af2e-175e-4af2-a5a4-144853f6af11-kube-api-access-8zm5b\") pod \"payload-processing-e2e-shared-a-w0-c42c0a-55d8fcbd6c-rg6mj\" (UID: \"bf53af2e-175e-4af2-a5a4-144853f6af11\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-c42c0a-55d8fcbd6c-rg6mj"
Oct 30 22:17:57.562367 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:57.540435    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf53af2e-175e-4af2-a5a4-144853f6af11-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-c42c0a-55d8fcbd6c-rg6mj\" (UID: \"bf53af2e-175e-4af2-a5a4-144853f6af11\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-c42c0a-55d8fcbd6c-rg6mj"
Oct 30 22:17:57.575822 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:57.575788    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-c42c0a-5694bffdff-4gb6s"]
Oct 30 22:17:57.609791 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:57.609479    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-c42c0a-5694bffdff-4gb6s"]
Oct 30 22:17:57.609791 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:57.609598    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-c42c0a-5694bffdff-4gb6s"
Oct 30 22:17:57.629827 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:57.629133    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zm5b\" (UniqueName: \"kubernetes.io/projected/bf53af2e-175e-4af2-a5a4-144853f6af11-kube-api-access-8zm5b\") pod \"payload-processing-e2e-shared-a-w0-c42c0a-55d8fcbd6c-rg6mj\" (UID: \"bf53af2e-175e-4af2-a5a4-144853f6af11\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-c42c0a-55d8fcbd6c-rg6mj"
Oct 30 22:17:57.629827 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:57.629382    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-c42c0a-55d8fcbd6c-rg6mj"
Oct 30 22:17:57.750256 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:57.748359    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7bwx\" (UniqueName: \"kubernetes.io/projected/195e9bc1-18c1-4fcc-9999-2518f332c8c5-kube-api-access-m7bwx\") pod \"payload-pre-processing-e2e-shared-a-w0-c42c0a-5694bffdff-4gb6s\" (UID: \"195e9bc1-18c1-4fcc-9999-2518f332c8c5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-c42c0a-5694bffdff-4gb6s"
Oct 30 22:17:57.750256 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:57.748471    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/195e9bc1-18c1-4fcc-9999-2518f332c8c5-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-c42c0a-5694bffdff-4gb6s\" (UID: \"195e9bc1-18c1-4fcc-9999-2518f332c8c5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-c42c0a-5694bffdff-4gb6s"
Oct 30 22:17:57.850814 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:57.850716    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7bwx\" (UniqueName: \"kubernetes.io/projected/195e9bc1-18c1-4fcc-9999-2518f332c8c5-kube-api-access-m7bwx\") pod \"payload-pre-processing-e2e-shared-a-w0-c42c0a-5694bffdff-4gb6s\" (UID: \"195e9bc1-18c1-4fcc-9999-2518f332c8c5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-c42c0a-5694bffdff-4gb6s"
Oct 30 22:17:57.850982 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:57.850825    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/195e9bc1-18c1-4fcc-9999-2518f332c8c5-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-c42c0a-5694bffdff-4gb6s\" (UID: \"195e9bc1-18c1-4fcc-9999-2518f332c8c5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-c42c0a-5694bffdff-4gb6s"
Oct 30 22:17:57.851611 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:57.851586    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/195e9bc1-18c1-4fcc-9999-2518f332c8c5-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-c42c0a-5694bffdff-4gb6s\" (UID: \"195e9bc1-18c1-4fcc-9999-2518f332c8c5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-c42c0a-5694bffdff-4gb6s"
Oct 30 22:17:57.910198 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:57.910160    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7bwx\" (UniqueName: \"kubernetes.io/projected/195e9bc1-18c1-4fcc-9999-2518f332c8c5-kube-api-access-m7bwx\") pod \"payload-pre-processing-e2e-shared-a-w0-c42c0a-5694bffdff-4gb6s\" (UID: \"195e9bc1-18c1-4fcc-9999-2518f332c8c5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-c42c0a-5694bffdff-4gb6s"
Oct 30 22:17:57.960114 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:57.960087    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-c42c0a-5694bffdff-4gb6s"
Oct 30 22:17:58.051505 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:58.051466    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-c42c0a-55d8fcbd6c-rg6mj"]
Oct 30 22:17:58.051879 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:17:58.051850    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf53af2e_175e_4af2_a5a4_144853f6af11.slice/crio-b83eec17b7630c730523c71ae20ddce3bdd8dfd36300caefd60100d401725d7f WatchSource:0}: Error finding container b83eec17b7630c730523c71ae20ddce3bdd8dfd36300caefd60100d401725d7f: Status 404 returned error can't find the container with id b83eec17b7630c730523c71ae20ddce3bdd8dfd36300caefd60100d401725d7f
Oct 30 22:17:58.344434 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:17:58.344387    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod195e9bc1_18c1_4fcc_9999_2518f332c8c5.slice/crio-c64e2e0884721a61260d75d3dfed21c66776059e5ae14b3aa70d19f48be105dd WatchSource:0}: Error finding container c64e2e0884721a61260d75d3dfed21c66776059e5ae14b3aa70d19f48be105dd: Status 404 returned error can't find the container with id c64e2e0884721a61260d75d3dfed21c66776059e5ae14b3aa70d19f48be105dd
Oct 30 22:17:58.348940 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:58.348915    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-c42c0a-5694bffdff-4gb6s"]
Oct 30 22:17:58.526755 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:58.526726    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="313acbe0-dc5c-48cb-92d8-972092227317" path="/var/lib/kubelet/pods/313acbe0-dc5c-48cb-92d8-972092227317/volumes"
Oct 30 22:17:58.585115 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:58.585081    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-c42c0a-55d8fcbd6c-rg6mj" event={"ID":"bf53af2e-175e-4af2-a5a4-144853f6af11","Type":"ContainerStarted","Data":"d3e4c658ab0439c22a104825bb226c4cf031bae0873256f67d10a4b84f6cf30b"}
Oct 30 22:17:58.585346 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:58.585142    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-c42c0a-55d8fcbd6c-rg6mj" event={"ID":"bf53af2e-175e-4af2-a5a4-144853f6af11","Type":"ContainerStarted","Data":"b83eec17b7630c730523c71ae20ddce3bdd8dfd36300caefd60100d401725d7f"}
Oct 30 22:17:58.585464 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:58.585434    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-c42c0a-55d8fcbd6c-rg6mj"
Oct 30 22:17:58.587131 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:58.587102    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-c42c0a-5694bffdff-4gb6s" event={"ID":"195e9bc1-18c1-4fcc-9999-2518f332c8c5","Type":"ContainerStarted","Data":"464bce49761620adf12fd89d4065c6d620555fc5ca4f81db9249bc1aaef59116"}
Oct 30 22:17:58.587254 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:58.587147    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-c42c0a-5694bffdff-4gb6s" event={"ID":"195e9bc1-18c1-4fcc-9999-2518f332c8c5","Type":"ContainerStarted","Data":"c64e2e0884721a61260d75d3dfed21c66776059e5ae14b3aa70d19f48be105dd"}
Oct 30 22:17:58.587254 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:58.587187    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-c42c0a-5694bffdff-4gb6s"
Oct 30 22:17:58.701699 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:58.701648    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-c42c0a-55d8fcbd6c-rg6mj" podStartSLOduration=1.701635145 podStartE2EDuration="1.701635145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:17:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:17:58.698756248 +0000 UTC m=+1656.303063812" watchObservedRunningTime="2026-10-30 22:17:58.701635145 +0000 UTC m=+1656.305942732"
Oct 30 22:17:58.839423 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:17:58.821989    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-c42c0a-5694bffdff-4gb6s" podStartSLOduration=1.82194842 podStartE2EDuration="1.82194842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:17:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:17:58.819527572 +0000 UTC m=+1656.423835126" watchObservedRunningTime="2026-10-30 22:17:58.82194842 +0000 UTC m=+1656.426255974"
Oct 30 22:18:02.410152 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:18:02.410071    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-bee2db53-544cd6f495-6xb8n"]
Oct 30 22:18:02.416048 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:18:02.416019    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-bee2db53-544cd6f495-6xb8n"
Oct 30 22:18:02.475552 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:18:02.475517    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-bee2db53-544cd6f495-6xb8n"]
Oct 30 22:18:02.530221 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:18:02.530186    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-bee2db53\""
Oct 30 22:18:02.557424 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:18:02.545047    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4hf9\" (UniqueName: \"kubernetes.io/projected/ef450e67-6b32-4617-a73d-c3d5ccbbf203-kube-api-access-q4hf9\") pod \"payload-pre-processing-e2e-derive-bee2db53-544cd6f495-6xb8n\" (UID: \"ef450e67-6b32-4617-a73d-c3d5ccbbf203\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-bee2db53-544cd6f495-6xb8n"
Oct 30 22:18:02.557424 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:18:02.545126    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef450e67-6b32-4617-a73d-c3d5ccbbf203-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-bee2db53-544cd6f495-6xb8n\" (UID: \"ef450e67-6b32-4617-a73d-c3d5ccbbf203\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-bee2db53-544cd6f495-6xb8n"
Oct 30 22:18:02.578589 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:18:02.578515    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-bee2db53-dockercfg-5rdsl\""
Oct 30 22:18:02.647338 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:18:02.647287    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4hf9\" (UniqueName: \"kubernetes.io/projected/ef450e67-6b32-4617-a73d-c3d5ccbbf203-kube-api-access-q4hf9\") pod \"payload-pre-processing-e2e-derive-bee2db53-544cd6f495-6xb8n\" (UID: \"ef450e67-6b32-4617-a73d-c3d5ccbbf203\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-bee2db53-544cd6f495-6xb8n"
Oct 30 22:18:02.647552 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:18:02.647370    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef450e67-6b32-4617-a73d-c3d5ccbbf203-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-bee2db53-544cd6f495-6xb8n\" (UID: \"ef450e67-6b32-4617-a73d-c3d5ccbbf203\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-bee2db53-544cd6f495-6xb8n"
Oct 30 22:18:02.648535 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:18:02.648507    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef450e67-6b32-4617-a73d-c3d5ccbbf203-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-bee2db53-544cd6f495-6xb8n\" (UID: \"ef450e67-6b32-4617-a73d-c3d5ccbbf203\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-bee2db53-544cd6f495-6xb8n"
Oct 30 22:18:02.708207 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:18:02.708133    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4hf9\" (UniqueName: \"kubernetes.io/projected/ef450e67-6b32-4617-a73d-c3d5ccbbf203-kube-api-access-q4hf9\") pod \"payload-pre-processing-e2e-derive-bee2db53-544cd6f495-6xb8n\" (UID: \"ef450e67-6b32-4617-a73d-c3d5ccbbf203\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-bee2db53-544cd6f495-6xb8n"
Oct 30 22:18:02.731891 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:18:02.731858    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-bee2db53-544cd6f495-6xb8n"
Oct 30 22:18:03.116600 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:18:03.116569    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef450e67_6b32_4617_a73d_c3d5ccbbf203.slice/crio-83d15cd43acdf8de4abec32a1cec16259c13ec668cf4ad8828be3a74778dff2f WatchSource:0}: Error finding container 83d15cd43acdf8de4abec32a1cec16259c13ec668cf4ad8828be3a74778dff2f: Status 404 returned error can't find the container with id 83d15cd43acdf8de4abec32a1cec16259c13ec668cf4ad8828be3a74778dff2f
Oct 30 22:18:03.117719 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:18:03.117695    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-bee2db53-544cd6f495-6xb8n"]
Oct 30 22:18:03.635002 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:18:03.634969    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-bee2db53-544cd6f495-6xb8n" event={"ID":"ef450e67-6b32-4617-a73d-c3d5ccbbf203","Type":"ContainerStarted","Data":"3176fa8e0acb06f0f3841c1397a05e625e59125ac83e31b9b1101ce61dd58691"}
Oct 30 22:18:03.635485 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:18:03.635020    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-bee2db53-544cd6f495-6xb8n" event={"ID":"ef450e67-6b32-4617-a73d-c3d5ccbbf203","Type":"ContainerStarted","Data":"83d15cd43acdf8de4abec32a1cec16259c13ec668cf4ad8828be3a74778dff2f"}
Oct 30 22:18:03.635485 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:18:03.635047    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-bee2db53-544cd6f495-6xb8n"
Oct 30 22:18:03.769554 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:18:03.769491    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-bee2db53-544cd6f495-6xb8n" podStartSLOduration=1.769476857 podStartE2EDuration="1.769476857s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:18:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:18:03.764054541 +0000 UTC m=+1661.368362096" watchObservedRunningTime="2026-10-30 22:18:03.769476857 +0000 UTC m=+1661.373784411"
Oct 30 22:18:05.385923 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:18:05.385892    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-88d9s"]
Oct 30 22:18:05.390653 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:18:05.390609    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-88d9s"
Oct 30 22:18:05.491259 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:18:05.491230    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-88d9s"]
Oct 30 22:18:05.501043 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:18:05.501008    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/76d2c82e-13ed-497c-acbc-7748bfb133c7-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-88d9s\" (UID: \"76d2c82e-13ed-497c-acbc-7748bfb133c7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-88d9s"
Oct 30 22:18:05.501177 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:18:05.501065    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/76d2c82e-13ed-497c-acbc-7748bfb133c7-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-88d9s\" (UID: \"76d2c82e-13ed-497c-acbc-7748bfb133c7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-88d9s"
Oct 30 22:18:05.501230 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:18:05.501201    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/76d2c82e-13ed-497c-acbc-7748bfb133c7-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-88d9s\" (UID: \"76d2c82e-13ed-497c-acbc-7748bfb133c7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-88d9s"
Oct 30 22:18:05.501296 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:18:05.501256    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76d2c82e-13ed-497c-acbc-7748bfb133c7-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-88d9s\" (UID: \"76d2c82e-13ed-497c-acbc-7748bfb133c7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-88d9s"
Oct 30 22:18:05.501445 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:18:05.501383    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jdz9\" (UniqueName: \"kubernetes.io/projected/76d2c82e-13ed-497c-acbc-7748bfb133c7-kube-api-access-2jdz9\") pod \"istiod-openshift-gateway-7ddd9cfdc8-88d9s\" (UID: \"76d2c82e-13ed-497c-acbc-7748bfb133c7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-88d9s"
Oct 30 22:18:05.501609 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:18:05.501590    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/76d2c82e-13ed-497c-acbc-7748bfb133c7-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-88d9s\" (UID: \"76d2c82e-13ed-497c-acbc-7748bfb133c7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-88d9s"
Oct 30 22:18:05.501654 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:18:05.501639    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/76d2c82e-13ed-497c-acbc-7748bfb133c7-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-88d9s\" (UID: \"76d2c82e-13ed-497c-acbc-7748bfb133c7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-88d9s"
Oct 30 22:18:05.603007 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:18:05.602973    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/76d2c82e-13ed-497c-acbc-7748bfb133c7-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-88d9s\" (UID: \"76d2c82e-13ed-497c-acbc-7748bfb133c7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-88d9s"
Oct 30 22:18:05.603165 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:18:05.603016    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/76d2c82e-13ed-497c-acbc-7748bfb133c7-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-88d9s\" (UID: \"76d2c82e-13ed-497c-acbc-7748bfb133c7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-88d9s"
Oct 30 22:18:05.603165 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:18:05.603056    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/76d2c82e-13ed-497c-acbc-7748bfb133c7-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-88d9s\" (UID: \"76d2c82e-13ed-497c-acbc-7748bfb133c7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-88d9s"
Oct 30 22:18:05.603165 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:18:05.603078    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76d2c82e-13ed-497c-acbc-7748bfb133c7-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-88d9s\" (UID: \"76d2c82e-13ed-497c-acbc-7748bfb133c7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-88d9s"
Oct 30 22:18:05.603165 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:18:05.603107    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jdz9\" (UniqueName: \"kubernetes.io/projected/76d2c82e-13ed-497c-acbc-7748bfb133c7-kube-api-access-2jdz9\") pod \"istiod-openshift-gateway-7ddd9cfdc8-88d9s\" (UID: \"76d2c82e-13ed-497c-acbc-7748bfb133c7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-88d9s"
Oct 30 22:18:05.603374 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:18:05.603243    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/76d2c82e-13ed-497c-acbc-7748bfb133c7-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-88d9s\" (UID: \"76d2c82e-13ed-497c-acbc-7748bfb133c7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-88d9s"
Oct 30 22:18:05.603374 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:18:05.603283    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/76d2c82e-13ed-497c-acbc-7748bfb133c7-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-88d9s\" (UID: \"76d2c82e-13ed-497c-acbc-7748bfb133c7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-88d9s"
Oct 30 22:18:05.603926 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:18:05.603899    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/76d2c82e-13ed-497c-acbc-7748bfb133c7-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-88d9s\" (UID: \"76d2c82e-13ed-497c-acbc-7748bfb133c7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-88d9s"
Oct 30 22:18:05.605563 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:18:05.605538    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/76d2c82e-13ed-497c-acbc-7748bfb133c7-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-88d9s\" (UID: \"76d2c82e-13ed-497c-acbc-7748bfb133c7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-88d9s"
Oct 30 22:18:05.605680 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:18:05.605605    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/76d2c82e-13ed-497c-acbc-7748bfb133c7-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-88d9s\" (UID: \"76d2c82e-13ed-497c-acbc-7748bfb133c7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-88d9s"
Oct 30 22:18:05.605680 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:18:05.605624    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/76d2c82e-13ed-497c-acbc-7748bfb133c7-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-88d9s\" (UID: \"76d2c82e-13ed-497c-acbc-7748bfb133c7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-88d9s"
Oct 30 22:18:05.605760 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:18:05.605687    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/76d2c82e-13ed-497c-acbc-7748bfb133c7-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-88d9s\" (UID: \"76d2c82e-13ed-497c-acbc-7748bfb133c7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-88d9s"
Oct 30 22:18:05.663549 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:18:05.663486    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76d2c82e-13ed-497c-acbc-7748bfb133c7-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-88d9s\" (UID: \"76d2c82e-13ed-497c-acbc-7748bfb133c7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-88d9s"
Oct 30 22:18:05.663688 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:18:05.663668    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jdz9\" (UniqueName: \"kubernetes.io/projected/76d2c82e-13ed-497c-acbc-7748bfb133c7-kube-api-access-2jdz9\") pod \"istiod-openshift-gateway-7ddd9cfdc8-88d9s\" (UID: \"76d2c82e-13ed-497c-acbc-7748bfb133c7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-88d9s"
Oct 30 22:18:05.708894 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:18:05.708862    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-88d9s"
Oct 30 22:18:06.163528 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:18:06.163494    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76d2c82e_13ed_497c_acbc_7748bfb133c7.slice/crio-7bbf67b43e173ffe70189ec8cf04b8d45d2e248025e4b04f9e69bdb4534dc788 WatchSource:0}: Error finding container 7bbf67b43e173ffe70189ec8cf04b8d45d2e248025e4b04f9e69bdb4534dc788: Status 404 returned error can't find the container with id 7bbf67b43e173ffe70189ec8cf04b8d45d2e248025e4b04f9e69bdb4534dc788
Oct 30 22:18:06.164334 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:18:06.164307    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-88d9s"]
Oct 30 22:18:06.166590 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:18:06.166551    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:18:06.166694 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:18:06.166627    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:18:06.661542 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:18:06.661507    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-88d9s" event={"ID":"76d2c82e-13ed-497c-acbc-7748bfb133c7","Type":"ContainerStarted","Data":"19944a2586d1a0f46f77065daa511ea819366a8510d92908d3a918724cf6a8f5"}
Oct 30 22:18:06.661946 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:18:06.661566    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-88d9s" event={"ID":"76d2c82e-13ed-497c-acbc-7748bfb133c7","Type":"ContainerStarted","Data":"7bbf67b43e173ffe70189ec8cf04b8d45d2e248025e4b04f9e69bdb4534dc788"}
Oct 30 22:18:06.661946 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:18:06.661697    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-88d9s"
Oct 30 22:18:06.793907 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:18:06.793846    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-88d9s" podStartSLOduration=1.793831876 podStartE2EDuration="1.793831876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:18:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:18:06.789849854 +0000 UTC m=+1664.394157404" watchObservedRunningTime="2026-10-30 22:18:06.793831876 +0000 UTC m=+1664.398139429"
Oct 30 22:18:07.698443 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:18:07.690472    5678 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-88d9s container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:18:07.698443 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:18:07.690526    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-88d9s" podUID="76d2c82e-13ed-497c-acbc-7748bfb133c7" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:18:09.065934 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:18:09.065897    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1515f79a-wn8hw"]
Oct 30 22:18:09.104972 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:18:09.104938    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1515f79a-wn8hw"]
Oct 30 22:18:09.598639 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:18:09.598611    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-c42c0a-55d8fcbd6c-rg6mj"
Oct 30 22:18:09.598836 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:18:09.598694    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-c42c0a-5694bffdff-4gb6s"
Oct 30 22:18:10.525292 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:18:10.525259    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24484902-a1b7-488b-bfcc-666e79cc88ee" path="/var/lib/kubelet/pods/24484902-a1b7-488b-bfcc-666e79cc88ee/volumes"
Oct 30 22:18:10.572322 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:18:10.572288    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 22:18:10.687192 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:18:10.687164    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-88d9s"
Oct 30 22:18:14.646543 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:18:14.646517    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-bee2db53-544cd6f495-6xb8n"
Oct 30 22:18:23.001682 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:18:23.001651    5678 scope.go:122] "RemoveContainer" containerID="3398adbaad00cbf191e24ca0549b8b2c6fc0a6524c1c6afb54b44f6b73851297"
Oct 30 22:18:23.013058 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:18:23.013039    5678 scope.go:122] "RemoveContainer" containerID="36b76e5664acf6be7359c5c8c453de6815ecb9e8ad2f45d70f7407bfdac8cecc"
Oct 30 22:18:35.077740 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:18:35.077701    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26472ab3-gr8x4"]
Oct 30 22:18:35.124759 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:18:35.124731    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26472ab3-gr8x4"]
Oct 30 22:18:36.527585 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:18:36.527545    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5cd4402-8612-4915-a5bd-debb68d321be" path="/var/lib/kubelet/pods/a5cd4402-8612-4915-a5bd-debb68d321be/volumes"
Oct 30 22:18:40.571622 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:18:40.571582    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 22:18:41.063273 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:18:41.063238    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2395b65-2ngnr"]
Oct 30 22:18:41.100572 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:18:41.100541    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2395b65-2ngnr"]
Oct 30 22:18:42.527520 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:18:42.527486    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c4f4212-b490-49a0-9e31-f74e31ec0f2f" path="/var/lib/kubelet/pods/9c4f4212-b490-49a0-9e31-f74e31ec0f2f/volumes"
Oct 30 22:18:52.031027 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:18:52.029112    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9ndd9" podUID="81bd6415-f42f-446b-a64e-956f794b3956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": context deadline exceeded"
Oct 30 22:18:52.125420 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:18:52.120514    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-65e59f/simulator-w4-65e59f-kserve-85698bcd47-przch" podUID="ee5e843b-17be-4623-8c95-dc07c8b7f506" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:18:52.340474 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:18:52.339441    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7bb1c2/distinct-w3-7bb1c2-kserve-8d695596c-8zgtw" podUID="36f0a6a2-54be-49bf-9202-502960cbb134" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:18:52.370051 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:18:52.366488    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7bb1c2/premium-w3-7bb1c2-kserve-7dbf875d88-dqxn5" podUID="a58c911f-e2eb-4820-a6b6-24afe78b5fd4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:18:52.370051 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:18:52.368159    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb0b93/simulator-w6-fb0b93-kserve-55b784b479-588bn" podUID="951a6f7e-a22e-40f2-ace2-1ce5d3bc9b49" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:18:52.447544 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:18:52.447496    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-65e59f/distinct-2-w4-65e59f-kserve-689566757d-bf98q" podUID="64e66fdb-056e-4086-a85a-8c02613584a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:18:52.448193 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:18:52.448163    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-65e59f/premium-w4-65e59f-kserve-6dc49468f8-rlzhx" podUID="8590a783-eb24-4759-ad74-1369f36aa9d2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded"
Oct 30 22:18:52.526773 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:18:52.526731    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a00d9b/distinct-2-w1-a00d9b-kserve-68fd95957c-dmgnb" podUID="23a81fda-f7e3-4f91-8d96-9f19c7de6e09" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:18:52.608091 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:18:52.607985    5678 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-7bb1c2/distinct-w3-7bb1c2-kserve-8d695596c-8zgtw" podUID="36f0a6a2-54be-49bf-9202-502960cbb134" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:18:52.790543 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:18:52.790493    5678 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-7bb1c2/simulator-w3-7bb1c2-kserve-675645869c-vcwqv" podUID="1fce9daf-68c9-41d0-825b-2012bb93c8ba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:18:52.791196 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:18:52.791166    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7bb1c2/simulator-w3-7bb1c2-kserve-675645869c-vcwqv" podUID="1fce9daf-68c9-41d0-825b-2012bb93c8ba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Oct 30 22:18:52.839550 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:18:52.839497    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb0b93/distinct-2-w6-fb0b93-kserve-7d65bb4698-m6tkh" podUID="6fd276ef-522b-4bb4-b101-0a9c3fe9ecd3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:18:52.946425 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:18:52.943263    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779f32/premium-w0-779f32-kserve-67978797c8-jpkvl" podUID="40138fb9-9a75-4458-a351-11eb61c4a1e3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:18:53.029559 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:18:53.029508    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9ndd9" podUID="81bd6415-f42f-446b-a64e-956f794b3956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:18:53.030839 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:18:53.030817    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9ndd9"
Oct 30 22:18:53.125918 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:18:53.125870    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-65e59f/simulator-w4-65e59f-kserve-85698bcd47-przch" podUID="ee5e843b-17be-4623-8c95-dc07c8b7f506" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Oct 30 22:18:53.127651 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:18:53.127062    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-65e59f/simulator-w4-65e59f-kserve-85698bcd47-przch"
Oct 30 22:18:53.469425 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:18:53.409605    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7bb1c2/distinct-w3-7bb1c2-kserve-8d695596c-8zgtw" podUID="36f0a6a2-54be-49bf-9202-502960cbb134" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded"
Oct 30 22:18:53.529711 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:18:53.529659    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7bb1c2/premium-w3-7bb1c2-kserve-7dbf875d88-dqxn5" podUID="a58c911f-e2eb-4820-a6b6-24afe78b5fd4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Oct 30 22:18:53.534519 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:18:53.534468    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-65e59f/premium-w4-65e59f-kserve-6dc49468f8-rlzhx" podUID="8590a783-eb24-4759-ad74-1369f36aa9d2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:18:53.534784 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:18:53.534741    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-65e59f/distinct-2-w4-65e59f-kserve-689566757d-bf98q" podUID="64e66fdb-056e-4086-a85a-8c02613584a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": context deadline exceeded"
Oct 30 22:18:53.607428 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:18:53.567664    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-7bb1c2/distinct-w3-7bb1c2-kserve-8d695596c-8zgtw"
Oct 30 22:18:53.882485 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:18:53.831484    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7bb1c2/simulator-w3-7bb1c2-kserve-675645869c-vcwqv" podUID="1fce9daf-68c9-41d0-825b-2012bb93c8ba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:18:53.882485 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:18:53.409876    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb0b93/simulator-w6-fb0b93-kserve-55b784b479-588bn" podUID="951a6f7e-a22e-40f2-ace2-1ce5d3bc9b49" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:18:53.882485 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:18:53.836846    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a00d9b/distinct-2-w1-a00d9b-kserve-68fd95957c-dmgnb" podUID="23a81fda-f7e3-4f91-8d96-9f19c7de6e09" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:18:53.882485 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:18:53.837007    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-7bb1c2/premium-w3-7bb1c2-kserve-7dbf875d88-dqxn5"
Oct 30 22:18:53.883713 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:18:53.883685    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-65e59f/premium-w4-65e59f-kserve-6dc49468f8-rlzhx"
Oct 30 22:18:53.883939 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:18:53.883924    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-65e59f/distinct-2-w4-65e59f-kserve-689566757d-bf98q"
Oct 30 22:18:53.884150 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:18:53.884135    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-7bb1c2/simulator-w3-7bb1c2-kserve-675645869c-vcwqv"
Oct 30 22:18:53.884290 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:18:53.884278    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-fb0b93/simulator-w6-fb0b93-kserve-55b784b479-588bn"
Oct 30 22:18:53.884432 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:18:53.884419    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a00d9b/distinct-2-w1-a00d9b-kserve-68fd95957c-dmgnb"
Oct 30 22:18:53.910810 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:18:53.909722    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb0b93/distinct-2-w6-fb0b93-kserve-7d65bb4698-m6tkh" podUID="6fd276ef-522b-4bb4-b101-0a9c3fe9ecd3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": context deadline exceeded"
Oct 30 22:18:53.910810 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:18:53.910572    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-fb0b93/distinct-2-w6-fb0b93-kserve-7d65bb4698-m6tkh"
Oct 30 22:18:54.035588 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:18:54.035540    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9ndd9" podUID="81bd6415-f42f-446b-a64e-956f794b3956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:18:54.198807 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:18:54.174481    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-65e59f/simulator-w4-65e59f-kserve-85698bcd47-przch" podUID="ee5e843b-17be-4623-8c95-dc07c8b7f506" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Oct 30 22:18:54.339558 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:18:54.339526    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a00d9b/distinct-2-w1-a00d9b-kserve-68fd95957c-dmgnb"
Oct 30 22:18:54.433424 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:18:54.409796    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-65e59f/simulator-w4-65e59f-kserve-85698bcd47-przch"
Oct 30 22:18:54.433424 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:18:54.431694    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-fb0b93/simulator-w6-fb0b93-kserve-55b784b479-588bn"
Oct 30 22:18:54.433424 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:18:54.431753    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9ndd9"
Oct 30 22:18:54.437630 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:18:54.437607    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-65e59f/premium-w4-65e59f-kserve-6dc49468f8-rlzhx"
Oct 30 22:18:54.474570 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:18:54.474471    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7bb1c2/distinct-w3-7bb1c2-kserve-8d695596c-8zgtw" podUID="36f0a6a2-54be-49bf-9202-502960cbb134" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:18:54.490925 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:18:54.490896    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-65e59f/distinct-2-w4-65e59f-kserve-689566757d-bf98q"
Oct 30 22:18:54.499442 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:18:54.491839    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-7bb1c2/simulator-w3-7bb1c2-kserve-675645869c-vcwqv"
Oct 30 22:18:54.592107 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:18:54.592083    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-7bb1c2/distinct-w3-7bb1c2-kserve-8d695596c-8zgtw"
Oct 30 22:18:54.592310 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:18:54.592300    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-fb0b93/distinct-2-w6-fb0b93-kserve-7d65bb4698-m6tkh"
Oct 30 22:18:54.592449 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:18:54.592439    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-7bb1c2/premium-w3-7bb1c2-kserve-7dbf875d88-dqxn5"
Oct 30 22:18:58.061311 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:18:58.061251    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a7082f2-b6vzv"]
Oct 30 22:18:58.107875 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:18:58.107842    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a7082f2-b6vzv"]
Oct 30 22:18:58.526760 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:18:58.526730    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3abb3e26-65da-49e8-85cf-7d0b01a1da7c" path="/var/lib/kubelet/pods/3abb3e26-65da-49e8-85cf-7d0b01a1da7c/volumes"
Oct 30 22:19:00.026910 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:19:00.026870    5678 generic.go:356] "Generic (PLEG): container finished" podID="669af24d-752d-407e-af9b-8ba5c02c40de" containerID="7ee0cb93980198060bf70153267f3892dc5b97235ca62c721304ba81388cf339" exitCode=0
Oct 30 22:19:00.027415 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:19:00.026942    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ac5024ad-openshift-default-65855d67d5-hpztt" event={"ID":"669af24d-752d-407e-af9b-8ba5c02c40de","Type":"ContainerDied","Data":"7ee0cb93980198060bf70153267f3892dc5b97235ca62c721304ba81388cf339"}
Oct 30 22:19:00.027496 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:19:00.027443    5678 scope.go:122] "RemoveContainer" containerID="7ee0cb93980198060bf70153267f3892dc5b97235ca62c721304ba81388cf339"
Oct 30 22:19:00.029158 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:19:00.029128    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:19:00.029245 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:19:00.029204    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:19:00.029245 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:19:00.029239    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:19:01.037060 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:19:01.037025    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ac5024ad-openshift-default-65855d67d5-hpztt" event={"ID":"669af24d-752d-407e-af9b-8ba5c02c40de","Type":"ContainerStarted","Data":"2799183d441628608760b7e2aed5fc7033e76f9cac49ac91728f334f9441d174"}
Oct 30 22:19:01.412691 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:19:01.412665    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-ac5024ad-openshift-default-65855d67d5-hpztt"
Oct 30 22:19:01.414041 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:19:01.414011    5678 patch_prober.go:29] interesting pod/e2e-ait-ac5024ad-openshift-default-65855d67d5-hpztt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body=
Oct 30 22:19:01.414163 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:19:01.414062    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ac5024ad-openshift-default-65855d67d5-hpztt" podUID="669af24d-752d-407e-af9b-8ba5c02c40de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused"
Oct 30 22:19:02.413220 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:19:02.413181    5678 patch_prober.go:29] interesting pod/e2e-ait-ac5024ad-openshift-default-65855d67d5-hpztt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body=
Oct 30 22:19:02.413630 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:19:02.413244    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ac5024ad-openshift-default-65855d67d5-hpztt" podUID="669af24d-752d-407e-af9b-8ba5c02c40de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused"
Oct 30 22:19:03.326438 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:19:03.325488    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-ac5024ad-openshift-default-65855d67d5-hpztt"
Oct 30 22:19:03.413573 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:19:03.413546    5678 patch_prober.go:29] interesting pod/e2e-ait-ac5024ad-openshift-default-65855d67d5-hpztt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body=
Oct 30 22:19:03.413960 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:19:03.413593    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ac5024ad-openshift-default-65855d67d5-hpztt" podUID="669af24d-752d-407e-af9b-8ba5c02c40de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused"
Oct 30 22:19:04.412953 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:19:04.412918    5678 patch_prober.go:29] interesting pod/e2e-ait-ac5024ad-openshift-default-65855d67d5-hpztt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body=
Oct 30 22:19:04.413131 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:19:04.412980    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ac5024ad-openshift-default-65855d67d5-hpztt" podUID="669af24d-752d-407e-af9b-8ba5c02c40de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused"
Oct 30 22:19:05.051660 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:19:05.051614    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d74eab4-zvgf6"]
Oct 30 22:19:05.056755 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:19:05.056727    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d74eab4-zvgf6"]
Oct 30 22:19:05.412735 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:19:05.412700    5678 patch_prober.go:29] interesting pod/e2e-ait-ac5024ad-openshift-default-65855d67d5-hpztt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body=
Oct 30 22:19:05.412908 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:19:05.412752    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ac5024ad-openshift-default-65855d67d5-hpztt" podUID="669af24d-752d-407e-af9b-8ba5c02c40de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused"
Oct 30 22:19:06.412829 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:19:06.412790    5678 patch_prober.go:29] interesting pod/e2e-ait-ac5024ad-openshift-default-65855d67d5-hpztt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body=
Oct 30 22:19:06.413226 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:19:06.412845    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ac5024ad-openshift-default-65855d67d5-hpztt" podUID="669af24d-752d-407e-af9b-8ba5c02c40de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused"
Oct 30 22:19:06.546424 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:19:06.540108    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c0aa3a7-b85d-4476-8cc1-73160b38374c" path="/var/lib/kubelet/pods/3c0aa3a7-b85d-4476-8cc1-73160b38374c/volumes"
Oct 30 22:19:07.417306 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:19:07.417277    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-ac5024ad-openshift-default-65855d67d5-hpztt"
Oct 30 22:19:07.418326 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:19:07.418305    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-ac5024ad-openshift-default-65855d67d5-hpztt"
Oct 30 22:19:10.574131 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:19:10.574094    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 22:19:11.051686 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:19:11.051649    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a05b2e8-shvt4"]
Oct 30 22:19:11.098599 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:19:11.098560    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a05b2e8-shvt4"]
Oct 30 22:19:12.524948 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:19:12.524916    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5152b232-a4a2-4b78-a6f9-f29695f54347" path="/var/lib/kubelet/pods/5152b232-a4a2-4b78-a6f9-f29695f54347/volumes"
Oct 30 22:19:16.057533 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:19:16.057497    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b531868-2xs6q"]
Oct 30 22:19:16.062364 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:19:16.062334    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b531868-2xs6q"]
Oct 30 22:19:16.527619 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:19:16.527583    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="839a167c-bed2-434e-8a2c-31635f2c467b" path="/var/lib/kubelet/pods/839a167c-bed2-434e-8a2c-31635f2c467b/volumes"
Oct 30 22:19:18.049231 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:19:18.049130    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-663bc197-khq9q"]
Oct 30 22:19:18.096057 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:19:18.096010    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-663bc197-khq9q"]
Oct 30 22:19:18.526130 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:19:18.526099    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60cf0c3e-6cb6-4090-8831-0ca156c59337" path="/var/lib/kubelet/pods/60cf0c3e-6cb6-4090-8831-0ca156c59337/volumes"
Oct 30 22:19:23.062980 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:19:23.062951    5678 scope.go:122] "RemoveContainer" containerID="87a3cab586aba9c2299193f92f3212b6ef910b5c1314a583a113ec14b83aaf46"
Oct 30 22:19:23.075576 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:19:23.075549    5678 scope.go:122] "RemoveContainer" containerID="0d50ec7ba68f313a6174199ad97d7a5732393f66af75db9ac9d5e5b029d9cf75"
Oct 30 22:19:23.085504 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:19:23.085482    5678 scope.go:122] "RemoveContainer" containerID="af848e644c9f307a49e16c1978ef8f5c3b042a5bd829c9195b1c5dbb18ad49fa"
Oct 30 22:19:27.080960 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:19:27.080928    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-58b3d74d-g984m"]
Oct 30 22:19:27.127763 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:19:27.127727    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-58b3d74d-g984m"]
Oct 30 22:19:28.524611 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:19:28.524579    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="638dfd81-12d9-438a-83d9-8b42fd7d5d0f" path="/var/lib/kubelet/pods/638dfd81-12d9-438a-83d9-8b42fd7d5d0f/volumes"
Oct 30 22:19:29.073881 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:19:29.073846    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3491e9e-jpg5w"]
Oct 30 22:19:29.120248 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:19:29.120221    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3491e9e-jpg5w"]
Oct 30 22:19:30.567488 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:19:30.567452    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9db99c04-1a54-413b-868b-196c86d4c074" path="/var/lib/kubelet/pods/9db99c04-1a54-413b-868b-196c86d4c074/volumes"
Oct 30 22:19:39.098828 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:19:39.098788    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3acbb058-68zvq"]
Oct 30 22:19:39.145456 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:19:39.145426    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3acbb058-68zvq"]
Oct 30 22:19:40.526416 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:19:40.526374    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="acfe6f2a-8f4e-4a7f-bde4-9486ed55cf07" path="/var/lib/kubelet/pods/acfe6f2a-8f4e-4a7f-bde4-9486ed55cf07/volumes"
Oct 30 22:19:40.574289 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:19:40.574244    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 22:19:45.081010 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:19:45.080933    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83996232-dxszx"]
Oct 30 22:19:45.127244 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:19:45.127216    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83996232-dxszx"]
Oct 30 22:19:46.613423 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:19:46.613334    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="506bf715-6464-4d0f-a5a7-85ad36039f21" path="/var/lib/kubelet/pods/506bf715-6464-4d0f-a5a7-85ad36039f21/volumes"
Oct 30 22:19:55.050099 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:19:55.050066    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53f4fd0e-rwrt9"]
Oct 30 22:19:55.096791 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:19:55.096762    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53f4fd0e-rwrt9"]
Oct 30 22:19:56.525468 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:19:56.525435    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0818834a-f7a8-4a39-97c4-159253f365df" path="/var/lib/kubelet/pods/0818834a-f7a8-4a39-97c4-159253f365df/volumes"
Oct 30 22:20:01.675307 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:01.675276    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-bee2db53-544cd6f495-6xb8n"]
Oct 30 22:20:02.684156 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:02.684094    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w5-3cc1b3-openshift-default-744b6b9d7f-lx8n5"]
Oct 30 22:20:02.689565 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:02.689544    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-3cc1b3-openshift-default-744b6b9d7f-lx8n5"
Oct 30 22:20:02.851816 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:02.851771    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b6efa36-7840-418b-be4a-723ca4407a91-istio-data\") pod \"e2e-shared-a-w5-3cc1b3-openshift-default-744b6b9d7f-lx8n5\" (UID: \"4b6efa36-7840-418b-be4a-723ca4407a91\") " pod="openshift-ingress/e2e-shared-a-w5-3cc1b3-openshift-default-744b6b9d7f-lx8n5"
Oct 30 22:20:02.852158 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:02.852118    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b6efa36-7840-418b-be4a-723ca4407a91-istio-podinfo\") pod \"e2e-shared-a-w5-3cc1b3-openshift-default-744b6b9d7f-lx8n5\" (UID: \"4b6efa36-7840-418b-be4a-723ca4407a91\") " pod="openshift-ingress/e2e-shared-a-w5-3cc1b3-openshift-default-744b6b9d7f-lx8n5"
Oct 30 22:20:02.852415 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:02.852372    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b6efa36-7840-418b-be4a-723ca4407a91-istio-token\") pod \"e2e-shared-a-w5-3cc1b3-openshift-default-744b6b9d7f-lx8n5\" (UID: \"4b6efa36-7840-418b-be4a-723ca4407a91\") " pod="openshift-ingress/e2e-shared-a-w5-3cc1b3-openshift-default-744b6b9d7f-lx8n5"
Oct 30 22:20:02.852582 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:02.852555    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b6efa36-7840-418b-be4a-723ca4407a91-istiod-ca-cert\") pod \"e2e-shared-a-w5-3cc1b3-openshift-default-744b6b9d7f-lx8n5\" (UID: \"4b6efa36-7840-418b-be4a-723ca4407a91\") " pod="openshift-ingress/e2e-shared-a-w5-3cc1b3-openshift-default-744b6b9d7f-lx8n5"
Oct 30 22:20:02.852825 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:02.852806    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b6efa36-7840-418b-be4a-723ca4407a91-istio-envoy\") pod \"e2e-shared-a-w5-3cc1b3-openshift-default-744b6b9d7f-lx8n5\" (UID: \"4b6efa36-7840-418b-be4a-723ca4407a91\") " pod="openshift-ingress/e2e-shared-a-w5-3cc1b3-openshift-default-744b6b9d7f-lx8n5"
Oct 30 22:20:02.852974 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:02.852955    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xwgg\" (UniqueName: \"kubernetes.io/projected/4b6efa36-7840-418b-be4a-723ca4407a91-kube-api-access-8xwgg\") pod \"e2e-shared-a-w5-3cc1b3-openshift-default-744b6b9d7f-lx8n5\" (UID: \"4b6efa36-7840-418b-be4a-723ca4407a91\") " pod="openshift-ingress/e2e-shared-a-w5-3cc1b3-openshift-default-744b6b9d7f-lx8n5"
Oct 30 22:20:02.860072 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:02.853116    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b6efa36-7840-418b-be4a-723ca4407a91-workload-certs\") pod \"e2e-shared-a-w5-3cc1b3-openshift-default-744b6b9d7f-lx8n5\" (UID: \"4b6efa36-7840-418b-be4a-723ca4407a91\") " pod="openshift-ingress/e2e-shared-a-w5-3cc1b3-openshift-default-744b6b9d7f-lx8n5"
Oct 30 22:20:02.860335 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:02.860309    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b6efa36-7840-418b-be4a-723ca4407a91-credential-socket\") pod \"e2e-shared-a-w5-3cc1b3-openshift-default-744b6b9d7f-lx8n5\" (UID: \"4b6efa36-7840-418b-be4a-723ca4407a91\") " pod="openshift-ingress/e2e-shared-a-w5-3cc1b3-openshift-default-744b6b9d7f-lx8n5"
Oct 30 22:20:02.860577 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:02.860556    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b6efa36-7840-418b-be4a-723ca4407a91-workload-socket\") pod \"e2e-shared-a-w5-3cc1b3-openshift-default-744b6b9d7f-lx8n5\" (UID: \"4b6efa36-7840-418b-be4a-723ca4407a91\") " pod="openshift-ingress/e2e-shared-a-w5-3cc1b3-openshift-default-744b6b9d7f-lx8n5"
Oct 30 22:20:02.870690 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:02.854780    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w5-3cc1b3-openshift-default-dockercfg-g7knk\""
Oct 30 22:20:02.900982 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:02.900936    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-3cc1b3-openshift-default-744b6b9d7f-lx8n5"]
Oct 30 22:20:02.976472 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:02.974935    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b6efa36-7840-418b-be4a-723ca4407a91-istio-envoy\") pod \"e2e-shared-a-w5-3cc1b3-openshift-default-744b6b9d7f-lx8n5\" (UID: \"4b6efa36-7840-418b-be4a-723ca4407a91\") " pod="openshift-ingress/e2e-shared-a-w5-3cc1b3-openshift-default-744b6b9d7f-lx8n5"
Oct 30 22:20:02.976472 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:02.974983    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xwgg\" (UniqueName: \"kubernetes.io/projected/4b6efa36-7840-418b-be4a-723ca4407a91-kube-api-access-8xwgg\") pod \"e2e-shared-a-w5-3cc1b3-openshift-default-744b6b9d7f-lx8n5\" (UID: \"4b6efa36-7840-418b-be4a-723ca4407a91\") " pod="openshift-ingress/e2e-shared-a-w5-3cc1b3-openshift-default-744b6b9d7f-lx8n5"
Oct 30 22:20:02.976472 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:02.975024    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b6efa36-7840-418b-be4a-723ca4407a91-workload-certs\") pod \"e2e-shared-a-w5-3cc1b3-openshift-default-744b6b9d7f-lx8n5\" (UID: \"4b6efa36-7840-418b-be4a-723ca4407a91\") " pod="openshift-ingress/e2e-shared-a-w5-3cc1b3-openshift-default-744b6b9d7f-lx8n5"
Oct 30 22:20:02.976472 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:02.975080    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b6efa36-7840-418b-be4a-723ca4407a91-credential-socket\") pod \"e2e-shared-a-w5-3cc1b3-openshift-default-744b6b9d7f-lx8n5\" (UID: \"4b6efa36-7840-418b-be4a-723ca4407a91\") " pod="openshift-ingress/e2e-shared-a-w5-3cc1b3-openshift-default-744b6b9d7f-lx8n5"
Oct 30 22:20:02.976472 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:02.975124    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b6efa36-7840-418b-be4a-723ca4407a91-workload-socket\") pod \"e2e-shared-a-w5-3cc1b3-openshift-default-744b6b9d7f-lx8n5\" (UID: \"4b6efa36-7840-418b-be4a-723ca4407a91\") " pod="openshift-ingress/e2e-shared-a-w5-3cc1b3-openshift-default-744b6b9d7f-lx8n5"
Oct 30 22:20:02.976472 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:02.975253    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b6efa36-7840-418b-be4a-723ca4407a91-istio-data\") pod \"e2e-shared-a-w5-3cc1b3-openshift-default-744b6b9d7f-lx8n5\" (UID: \"4b6efa36-7840-418b-be4a-723ca4407a91\") " pod="openshift-ingress/e2e-shared-a-w5-3cc1b3-openshift-default-744b6b9d7f-lx8n5"
Oct 30 22:20:02.976472 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:02.975288    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b6efa36-7840-418b-be4a-723ca4407a91-istio-podinfo\") pod \"e2e-shared-a-w5-3cc1b3-openshift-default-744b6b9d7f-lx8n5\" (UID: \"4b6efa36-7840-418b-be4a-723ca4407a91\") " pod="openshift-ingress/e2e-shared-a-w5-3cc1b3-openshift-default-744b6b9d7f-lx8n5"
Oct 30 22:20:02.976472 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:02.975327    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b6efa36-7840-418b-be4a-723ca4407a91-istio-token\") pod \"e2e-shared-a-w5-3cc1b3-openshift-default-744b6b9d7f-lx8n5\" (UID: \"4b6efa36-7840-418b-be4a-723ca4407a91\") " pod="openshift-ingress/e2e-shared-a-w5-3cc1b3-openshift-default-744b6b9d7f-lx8n5"
Oct 30 22:20:02.976472 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:02.975344    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b6efa36-7840-418b-be4a-723ca4407a91-istiod-ca-cert\") pod \"e2e-shared-a-w5-3cc1b3-openshift-default-744b6b9d7f-lx8n5\" (UID: \"4b6efa36-7840-418b-be4a-723ca4407a91\") " pod="openshift-ingress/e2e-shared-a-w5-3cc1b3-openshift-default-744b6b9d7f-lx8n5"
Oct 30 22:20:02.976472 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:02.976110    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b6efa36-7840-418b-be4a-723ca4407a91-istiod-ca-cert\") pod \"e2e-shared-a-w5-3cc1b3-openshift-default-744b6b9d7f-lx8n5\" (UID: \"4b6efa36-7840-418b-be4a-723ca4407a91\") " pod="openshift-ingress/e2e-shared-a-w5-3cc1b3-openshift-default-744b6b9d7f-lx8n5"
Oct 30 22:20:02.977394 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:02.977369    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b6efa36-7840-418b-be4a-723ca4407a91-workload-socket\") pod \"e2e-shared-a-w5-3cc1b3-openshift-default-744b6b9d7f-lx8n5\" (UID: \"4b6efa36-7840-418b-be4a-723ca4407a91\") " pod="openshift-ingress/e2e-shared-a-w5-3cc1b3-openshift-default-744b6b9d7f-lx8n5"
Oct 30 22:20:02.978070 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:02.977925    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b6efa36-7840-418b-be4a-723ca4407a91-workload-certs\") pod \"e2e-shared-a-w5-3cc1b3-openshift-default-744b6b9d7f-lx8n5\" (UID: \"4b6efa36-7840-418b-be4a-723ca4407a91\") " pod="openshift-ingress/e2e-shared-a-w5-3cc1b3-openshift-default-744b6b9d7f-lx8n5"
Oct 30 22:20:02.979512 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:02.978908    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b6efa36-7840-418b-be4a-723ca4407a91-istio-data\") pod \"e2e-shared-a-w5-3cc1b3-openshift-default-744b6b9d7f-lx8n5\" (UID: \"4b6efa36-7840-418b-be4a-723ca4407a91\") " pod="openshift-ingress/e2e-shared-a-w5-3cc1b3-openshift-default-744b6b9d7f-lx8n5"
Oct 30 22:20:02.979609 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:02.979125    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b6efa36-7840-418b-be4a-723ca4407a91-credential-socket\") pod \"e2e-shared-a-w5-3cc1b3-openshift-default-744b6b9d7f-lx8n5\" (UID: \"4b6efa36-7840-418b-be4a-723ca4407a91\") " pod="openshift-ingress/e2e-shared-a-w5-3cc1b3-openshift-default-744b6b9d7f-lx8n5"
Oct 30 22:20:02.991040 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:02.990630    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b6efa36-7840-418b-be4a-723ca4407a91-istio-podinfo\") pod \"e2e-shared-a-w5-3cc1b3-openshift-default-744b6b9d7f-lx8n5\" (UID: \"4b6efa36-7840-418b-be4a-723ca4407a91\") " pod="openshift-ingress/e2e-shared-a-w5-3cc1b3-openshift-default-744b6b9d7f-lx8n5"
Oct 30 22:20:02.991040 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:02.990928    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b6efa36-7840-418b-be4a-723ca4407a91-istio-envoy\") pod \"e2e-shared-a-w5-3cc1b3-openshift-default-744b6b9d7f-lx8n5\" (UID: \"4b6efa36-7840-418b-be4a-723ca4407a91\") " pod="openshift-ingress/e2e-shared-a-w5-3cc1b3-openshift-default-744b6b9d7f-lx8n5"
Oct 30 22:20:03.037216 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:03.037183    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xwgg\" (UniqueName: \"kubernetes.io/projected/4b6efa36-7840-418b-be4a-723ca4407a91-kube-api-access-8xwgg\") pod \"e2e-shared-a-w5-3cc1b3-openshift-default-744b6b9d7f-lx8n5\" (UID: \"4b6efa36-7840-418b-be4a-723ca4407a91\") " pod="openshift-ingress/e2e-shared-a-w5-3cc1b3-openshift-default-744b6b9d7f-lx8n5"
Oct 30 22:20:03.039127 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:03.039101    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b6efa36-7840-418b-be4a-723ca4407a91-istio-token\") pod \"e2e-shared-a-w5-3cc1b3-openshift-default-744b6b9d7f-lx8n5\" (UID: \"4b6efa36-7840-418b-be4a-723ca4407a91\") " pod="openshift-ingress/e2e-shared-a-w5-3cc1b3-openshift-default-744b6b9d7f-lx8n5"
Oct 30 22:20:03.306505 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:03.306385    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-3cc1b3-openshift-default-744b6b9d7f-lx8n5"
Oct 30 22:20:03.817153 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:20:03.817121    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b6efa36_7840_418b_be4a_723ca4407a91.slice/crio-fb6aa72538d513818b553307983046e93fe1c0731ac3a321ebdb8cceb3826464 WatchSource:0}: Error finding container fb6aa72538d513818b553307983046e93fe1c0731ac3a321ebdb8cceb3826464: Status 404 returned error can't find the container with id fb6aa72538d513818b553307983046e93fe1c0731ac3a321ebdb8cceb3826464
Oct 30 22:20:03.817653 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:03.817532    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-3cc1b3-openshift-default-744b6b9d7f-lx8n5"]
Oct 30 22:20:03.820215 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:03.820179    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:20:03.820350 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:03.820244    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:20:03.820350 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:03.820300    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:20:04.717309 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:04.717275    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-3cc1b3-openshift-default-744b6b9d7f-lx8n5" event={"ID":"4b6efa36-7840-418b-be4a-723ca4407a91","Type":"ContainerStarted","Data":"0f413616c94f35de7224875324a6cc2aa3bc8255d48c60b79f78ca0fafddd51d"}
Oct 30 22:20:04.717516 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:04.717335    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-3cc1b3-openshift-default-744b6b9d7f-lx8n5" event={"ID":"4b6efa36-7840-418b-be4a-723ca4407a91","Type":"ContainerStarted","Data":"fb6aa72538d513818b553307983046e93fe1c0731ac3a321ebdb8cceb3826464"}
Oct 30 22:20:04.883378 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:04.883319    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w5-3cc1b3-openshift-default-744b6b9d7f-lx8n5" podStartSLOduration=2.8833003550000003 podStartE2EDuration="2.883300355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:20:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:20:04.879089972 +0000 UTC m=+1782.483397527" watchObservedRunningTime="2026-10-30 22:20:04.883300355 +0000 UTC m=+1782.487607907"
Oct 30 22:20:05.306879 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:05.306849    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-3cc1b3-openshift-default-744b6b9d7f-lx8n5"
Oct 30 22:20:05.311919 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:05.311891    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-3cc1b3-openshift-default-744b6b9d7f-lx8n5"
Oct 30 22:20:05.724846 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:05.724815    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-3cc1b3-openshift-default-744b6b9d7f-lx8n5"
Oct 30 22:20:05.725822 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:05.725805    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-3cc1b3-openshift-default-744b6b9d7f-lx8n5"
Oct 30 22:20:06.676279 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:06.676225    5678 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-bee2db53-544cd6f495-6xb8n" podUID="ef450e67-6b32-4617-a73d-c3d5ccbbf203" containerName="payload-pre-processing" containerID="cri-o://3176fa8e0acb06f0f3841c1397a05e625e59125ac83e31b9b1101ce61dd58691" gracePeriod=25
Oct 30 22:20:06.961785 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:06.961757    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-bee2db53-544cd6f495-6xb8n"
Oct 30 22:20:07.054805 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:07.054777    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef450e67-6b32-4617-a73d-c3d5ccbbf203-kube-api-access-q4hf9\" (UniqueName: \"kubernetes.io/projected/ef450e67-6b32-4617-a73d-c3d5ccbbf203-kube-api-access-q4hf9\") pod \"ef450e67-6b32-4617-a73d-c3d5ccbbf203\" (UID: \"ef450e67-6b32-4617-a73d-c3d5ccbbf203\") "
Oct 30 22:20:07.054985 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:07.054818    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef450e67-6b32-4617-a73d-c3d5ccbbf203-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef450e67-6b32-4617-a73d-c3d5ccbbf203-plugins-config-volume\") pod \"ef450e67-6b32-4617-a73d-c3d5ccbbf203\" (UID: \"ef450e67-6b32-4617-a73d-c3d5ccbbf203\") "
Oct 30 22:20:07.055235 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:07.055207    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef450e67-6b32-4617-a73d-c3d5ccbbf203-plugins-config-volume" pod "ef450e67-6b32-4617-a73d-c3d5ccbbf203" (UID: "ef450e67-6b32-4617-a73d-c3d5ccbbf203"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:20:07.055549 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:07.055531    5678 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef450e67-6b32-4617-a73d-c3d5ccbbf203-plugins-config-volume\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:20:07.056840 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:07.056815    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef450e67-6b32-4617-a73d-c3d5ccbbf203-kube-api-access-q4hf9" pod "ef450e67-6b32-4617-a73d-c3d5ccbbf203" (UID: "ef450e67-6b32-4617-a73d-c3d5ccbbf203"). InnerVolumeSpecName "kube-api-access-q4hf9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:20:07.157590 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:07.157552    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4hf9\" (UniqueName: \"kubernetes.io/projected/ef450e67-6b32-4617-a73d-c3d5ccbbf203-kube-api-access-q4hf9\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:20:07.741008 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:07.740970    5678 generic.go:356] "Generic (PLEG): container finished" podID="ef450e67-6b32-4617-a73d-c3d5ccbbf203" containerID="3176fa8e0acb06f0f3841c1397a05e625e59125ac83e31b9b1101ce61dd58691" exitCode=0
Oct 30 22:20:07.741008 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:07.741007    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-bee2db53-544cd6f495-6xb8n"
Oct 30 22:20:07.741676 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:07.741054    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-bee2db53-544cd6f495-6xb8n" event={"ID":"ef450e67-6b32-4617-a73d-c3d5ccbbf203","Type":"ContainerDied","Data":"3176fa8e0acb06f0f3841c1397a05e625e59125ac83e31b9b1101ce61dd58691"}
Oct 30 22:20:07.741676 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:07.741121    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-bee2db53-544cd6f495-6xb8n" event={"ID":"ef450e67-6b32-4617-a73d-c3d5ccbbf203","Type":"ContainerDied","Data":"83d15cd43acdf8de4abec32a1cec16259c13ec668cf4ad8828be3a74778dff2f"}
Oct 30 22:20:07.741676 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:07.741152    5678 scope.go:122] "RemoveContainer" containerID="3176fa8e0acb06f0f3841c1397a05e625e59125ac83e31b9b1101ce61dd58691"
Oct 30 22:20:07.753811 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:07.753796    5678 scope.go:122] "RemoveContainer" containerID="3176fa8e0acb06f0f3841c1397a05e625e59125ac83e31b9b1101ce61dd58691"
Oct 30 22:20:07.754080 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:20:07.754065    5678 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3176fa8e0acb06f0f3841c1397a05e625e59125ac83e31b9b1101ce61dd58691\": container with ID starting with 3176fa8e0acb06f0f3841c1397a05e625e59125ac83e31b9b1101ce61dd58691 not found: ID does not exist" containerID="3176fa8e0acb06f0f3841c1397a05e625e59125ac83e31b9b1101ce61dd58691"
Oct 30 22:20:07.754130 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:07.754089    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3176fa8e0acb06f0f3841c1397a05e625e59125ac83e31b9b1101ce61dd58691"} err="failed to get container status \"3176fa8e0acb06f0f3841c1397a05e625e59125ac83e31b9b1101ce61dd58691\": rpc error: code = NotFound desc = could not find container \"3176fa8e0acb06f0f3841c1397a05e625e59125ac83e31b9b1101ce61dd58691\": container with ID starting with 3176fa8e0acb06f0f3841c1397a05e625e59125ac83e31b9b1101ce61dd58691 not found: ID does not exist"
Oct 30 22:20:07.916497 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:07.916463    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-bee2db53-544cd6f495-6xb8n"]
Oct 30 22:20:07.965141 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:07.965101    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-bee2db53-544cd6f495-6xb8n"]
Oct 30 22:20:08.526236 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:08.526200    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef450e67-6b32-4617-a73d-c3d5ccbbf203" path="/var/lib/kubelet/pods/ef450e67-6b32-4617-a73d-c3d5ccbbf203/volumes"
Oct 30 22:20:10.574472 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:20:10.574434    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 22:20:23.148974 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:23.148857    5678 scope.go:122] "RemoveContainer" containerID="4648bdf888daf6fc5d53e9b15701b7474718bebb0e66284398db0e9addff264e"
Oct 30 22:20:23.185575 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:23.185549    5678 scope.go:122] "RemoveContainer" containerID="eea81a21b0822262309c1bf7bb93cd6f8789330db8428dc08f7d202ee687bb2f"
Oct 30 22:20:23.234934 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:23.234906    5678 scope.go:122] "RemoveContainer" containerID="159b56382d26b5e30f19e0c5d18735482fa9ebcc05d20397a7dd897521fbfdfa"
Oct 30 22:20:23.254837 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:23.254009    5678 scope.go:122] "RemoveContainer" containerID="6718a3fd2b71806234a6062c4f2b8773b433b276151b2989256274d61c519c9b"
Oct 30 22:20:23.281084 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:23.281063    5678 scope.go:122] "RemoveContainer" containerID="3646900a47069009182de0f1460707bc1eac9d55cf690cd9925e764a6162c31f"
Oct 30 22:20:23.304630 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:23.293727    5678 scope.go:122] "RemoveContainer" containerID="9a25a0a5ed3ed0a9868be33935fc6b2a3f37167675aa545218d996929a1ad98e"
Oct 30 22:20:23.312625 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:23.312604    5678 scope.go:122] "RemoveContainer" containerID="77a5e895a63aac5e6945085eed7300922f6d097ab82a18a3b4123eb52a737abd"
Oct 30 22:20:25.903504 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:25.897173    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-ac5024ad-openshift-default-65855d67d5-hpztt"]
Oct 30 22:20:25.903504 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:25.897480    5678 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-ac5024ad-openshift-default-65855d67d5-hpztt" podUID="669af24d-752d-407e-af9b-8ba5c02c40de" containerName="istio-proxy" containerID="cri-o://2799183d441628608760b7e2aed5fc7033e76f9cac49ac91728f334f9441d174" gracePeriod=30
Oct 30 22:20:26.095426 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:26.094594    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f42d803d-drk78"]
Oct 30 22:20:26.113420 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:26.111269    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f42d803d-drk78"]
Oct 30 22:20:26.524831 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:26.524799    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05eb2ec8-ad05-45bb-98a6-c074474fab94" path="/var/lib/kubelet/pods/05eb2ec8-ad05-45bb-98a6-c074474fab94/volumes"
Oct 30 22:20:31.219819 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:31.219794    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ac5024ad-openshift-default-65855d67d5-hpztt"
Oct 30 22:20:31.246119 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:31.246093    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/669af24d-752d-407e-af9b-8ba5c02c40de-kube-api-access-b2jkv\" (UniqueName: \"kubernetes.io/projected/669af24d-752d-407e-af9b-8ba5c02c40de-kube-api-access-b2jkv\") pod \"669af24d-752d-407e-af9b-8ba5c02c40de\" (UID: \"669af24d-752d-407e-af9b-8ba5c02c40de\") "
Oct 30 22:20:31.246225 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:31.246135    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/669af24d-752d-407e-af9b-8ba5c02c40de-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/669af24d-752d-407e-af9b-8ba5c02c40de-istio-data\") pod \"669af24d-752d-407e-af9b-8ba5c02c40de\" (UID: \"669af24d-752d-407e-af9b-8ba5c02c40de\") "
Oct 30 22:20:31.246301 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:31.246279    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/669af24d-752d-407e-af9b-8ba5c02c40de-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/669af24d-752d-407e-af9b-8ba5c02c40de-workload-certs\") pod \"669af24d-752d-407e-af9b-8ba5c02c40de\" (UID: \"669af24d-752d-407e-af9b-8ba5c02c40de\") "
Oct 30 22:20:31.246359 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:31.246333    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/669af24d-752d-407e-af9b-8ba5c02c40de-istio-data" pod "669af24d-752d-407e-af9b-8ba5c02c40de" (UID: "669af24d-752d-407e-af9b-8ba5c02c40de"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:20:31.246359 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:31.246339    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/669af24d-752d-407e-af9b-8ba5c02c40de-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/669af24d-752d-407e-af9b-8ba5c02c40de-istiod-ca-cert\") pod \"669af24d-752d-407e-af9b-8ba5c02c40de\" (UID: \"669af24d-752d-407e-af9b-8ba5c02c40de\") "
Oct 30 22:20:31.246515 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:31.246472    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/669af24d-752d-407e-af9b-8ba5c02c40de-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/669af24d-752d-407e-af9b-8ba5c02c40de-credential-socket\") pod \"669af24d-752d-407e-af9b-8ba5c02c40de\" (UID: \"669af24d-752d-407e-af9b-8ba5c02c40de\") "
Oct 30 22:20:31.246587 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:31.246559    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/669af24d-752d-407e-af9b-8ba5c02c40de-workload-certs" pod "669af24d-752d-407e-af9b-8ba5c02c40de" (UID: "669af24d-752d-407e-af9b-8ba5c02c40de"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:20:31.246648 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:31.246565    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/669af24d-752d-407e-af9b-8ba5c02c40de-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/669af24d-752d-407e-af9b-8ba5c02c40de-istio-envoy\") pod \"669af24d-752d-407e-af9b-8ba5c02c40de\" (UID: \"669af24d-752d-407e-af9b-8ba5c02c40de\") "
Oct 30 22:20:31.246794 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:31.246766    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/669af24d-752d-407e-af9b-8ba5c02c40de-credential-socket" pod "669af24d-752d-407e-af9b-8ba5c02c40de" (UID: "669af24d-752d-407e-af9b-8ba5c02c40de"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:20:31.246794 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:31.246789    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/669af24d-752d-407e-af9b-8ba5c02c40de-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/669af24d-752d-407e-af9b-8ba5c02c40de-workload-socket\") pod \"669af24d-752d-407e-af9b-8ba5c02c40de\" (UID: \"669af24d-752d-407e-af9b-8ba5c02c40de\") "
Oct 30 22:20:31.246984 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:31.246865    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/669af24d-752d-407e-af9b-8ba5c02c40de-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/669af24d-752d-407e-af9b-8ba5c02c40de-istio-podinfo\") pod \"669af24d-752d-407e-af9b-8ba5c02c40de\" (UID: \"669af24d-752d-407e-af9b-8ba5c02c40de\") "
Oct 30 22:20:31.246984 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:31.246937    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/669af24d-752d-407e-af9b-8ba5c02c40de-istio-token\" (UniqueName: \"kubernetes.io/projected/669af24d-752d-407e-af9b-8ba5c02c40de-istio-token\") pod \"669af24d-752d-407e-af9b-8ba5c02c40de\" (UID: \"669af24d-752d-407e-af9b-8ba5c02c40de\") "
Oct 30 22:20:31.247084 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:31.246687    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/669af24d-752d-407e-af9b-8ba5c02c40de-istiod-ca-cert" pod "669af24d-752d-407e-af9b-8ba5c02c40de" (UID: "669af24d-752d-407e-af9b-8ba5c02c40de"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:20:31.247084 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:31.246976    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/669af24d-752d-407e-af9b-8ba5c02c40de-workload-socket" pod "669af24d-752d-407e-af9b-8ba5c02c40de" (UID: "669af24d-752d-407e-af9b-8ba5c02c40de"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:20:31.247576 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:31.247523    5678 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/669af24d-752d-407e-af9b-8ba5c02c40de-istio-data\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:20:31.247576 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:31.247568    5678 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/669af24d-752d-407e-af9b-8ba5c02c40de-workload-certs\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:20:31.247727 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:31.247585    5678 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/669af24d-752d-407e-af9b-8ba5c02c40de-istiod-ca-cert\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:20:31.247727 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:31.247602    5678 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/669af24d-752d-407e-af9b-8ba5c02c40de-credential-socket\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:20:31.247727 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:31.247614    5678 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/669af24d-752d-407e-af9b-8ba5c02c40de-workload-socket\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:20:31.248696 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:31.248674    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/669af24d-752d-407e-af9b-8ba5c02c40de-kube-api-access-b2jkv" pod "669af24d-752d-407e-af9b-8ba5c02c40de" (UID: "669af24d-752d-407e-af9b-8ba5c02c40de"). InnerVolumeSpecName "kube-api-access-b2jkv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:20:31.249111 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:31.249097    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/669af24d-752d-407e-af9b-8ba5c02c40de-istio-envoy" pod "669af24d-752d-407e-af9b-8ba5c02c40de" (UID: "669af24d-752d-407e-af9b-8ba5c02c40de"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:20:31.249250 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:31.249230    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/669af24d-752d-407e-af9b-8ba5c02c40de-istio-token" pod "669af24d-752d-407e-af9b-8ba5c02c40de" (UID: "669af24d-752d-407e-af9b-8ba5c02c40de"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:20:31.249329 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:31.249249    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/669af24d-752d-407e-af9b-8ba5c02c40de-istio-podinfo" pod "669af24d-752d-407e-af9b-8ba5c02c40de" (UID: "669af24d-752d-407e-af9b-8ba5c02c40de"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:20:31.349835 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:31.349795    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2jkv\" (UniqueName: \"kubernetes.io/projected/669af24d-752d-407e-af9b-8ba5c02c40de-kube-api-access-b2jkv\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:20:31.349835 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:31.349831    5678 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/669af24d-752d-407e-af9b-8ba5c02c40de-istio-envoy\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:20:31.350073 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:31.349845    5678 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/669af24d-752d-407e-af9b-8ba5c02c40de-istio-podinfo\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:20:31.350073 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:31.349858    5678 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/669af24d-752d-407e-af9b-8ba5c02c40de-istio-token\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:20:31.993957 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:31.993922    5678 generic.go:356] "Generic (PLEG): container finished" podID="669af24d-752d-407e-af9b-8ba5c02c40de" containerID="2799183d441628608760b7e2aed5fc7033e76f9cac49ac91728f334f9441d174" exitCode=0
Oct 30 22:20:31.994136 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:31.993982    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ac5024ad-openshift-default-65855d67d5-hpztt"
Oct 30 22:20:31.994136 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:31.993984    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ac5024ad-openshift-default-65855d67d5-hpztt" event={"ID":"669af24d-752d-407e-af9b-8ba5c02c40de","Type":"ContainerDied","Data":"2799183d441628608760b7e2aed5fc7033e76f9cac49ac91728f334f9441d174"}
Oct 30 22:20:31.994136 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:31.994127    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ac5024ad-openshift-default-65855d67d5-hpztt" event={"ID":"669af24d-752d-407e-af9b-8ba5c02c40de","Type":"ContainerDied","Data":"8e58a9f2f952413640658623bd116f8b10951368f20c9521f0876e68373eaaa1"}
Oct 30 22:20:31.994243 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:31.994156    5678 scope.go:122] "RemoveContainer" containerID="2799183d441628608760b7e2aed5fc7033e76f9cac49ac91728f334f9441d174"
Oct 30 22:20:32.009580 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:32.009559    5678 scope.go:122] "RemoveContainer" containerID="7ee0cb93980198060bf70153267f3892dc5b97235ca62c721304ba81388cf339"
Oct 30 22:20:32.020269 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:32.020099    5678 scope.go:122] "RemoveContainer" containerID="2799183d441628608760b7e2aed5fc7033e76f9cac49ac91728f334f9441d174"
Oct 30 22:20:32.020394 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:20:32.020373    5678 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2799183d441628608760b7e2aed5fc7033e76f9cac49ac91728f334f9441d174\": container with ID starting with 2799183d441628608760b7e2aed5fc7033e76f9cac49ac91728f334f9441d174 not found: ID does not exist" containerID="2799183d441628608760b7e2aed5fc7033e76f9cac49ac91728f334f9441d174"
Oct 30 22:20:32.020503 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:32.020419    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2799183d441628608760b7e2aed5fc7033e76f9cac49ac91728f334f9441d174"} err="failed to get container status \"2799183d441628608760b7e2aed5fc7033e76f9cac49ac91728f334f9441d174\": rpc error: code = NotFound desc = could not find container \"2799183d441628608760b7e2aed5fc7033e76f9cac49ac91728f334f9441d174\": container with ID starting with 2799183d441628608760b7e2aed5fc7033e76f9cac49ac91728f334f9441d174 not found: ID does not exist"
Oct 30 22:20:32.020503 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:32.020448    5678 scope.go:122] "RemoveContainer" containerID="7ee0cb93980198060bf70153267f3892dc5b97235ca62c721304ba81388cf339"
Oct 30 22:20:32.020736 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:20:32.020720    5678 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ee0cb93980198060bf70153267f3892dc5b97235ca62c721304ba81388cf339\": container with ID starting with 7ee0cb93980198060bf70153267f3892dc5b97235ca62c721304ba81388cf339 not found: ID does not exist" containerID="7ee0cb93980198060bf70153267f3892dc5b97235ca62c721304ba81388cf339"
Oct 30 22:20:32.020791 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:32.020743    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ee0cb93980198060bf70153267f3892dc5b97235ca62c721304ba81388cf339"} err="failed to get container status \"7ee0cb93980198060bf70153267f3892dc5b97235ca62c721304ba81388cf339\": rpc error: code = NotFound desc = could not find container \"7ee0cb93980198060bf70153267f3892dc5b97235ca62c721304ba81388cf339\": container with ID starting with 7ee0cb93980198060bf70153267f3892dc5b97235ca62c721304ba81388cf339 not found: ID does not exist"
Oct 30 22:20:32.203656 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:32.203626    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-ac5024ad-openshift-default-65855d67d5-hpztt"]
Oct 30 22:20:32.208569 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:32.208544    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-ac5024ad-openshift-default-65855d67d5-hpztt"]
Oct 30 22:20:32.524222 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:32.524192    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="669af24d-752d-407e-af9b-8ba5c02c40de" path="/var/lib/kubelet/pods/669af24d-752d-407e-af9b-8ba5c02c40de/volumes"
Oct 30 22:20:36.533523 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:36.533483    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-f2qb5"]
Oct 30 22:20:36.534102 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:36.533825    5678 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-f2qb5" podUID="5041edee-ad23-43bf-a84e-9bb6632c2c0b" containerName="istio-proxy" containerID="cri-o://25ad9f97a8fe5ee3c92c23e1e4730013643b77854b77e606209453c96d9cd173" gracePeriod=30
Oct 30 22:20:36.560250 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:36.559329    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-m2jqj"]
Oct 30 22:20:36.560250 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:36.559613    5678 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-m2jqj" podUID="06ae4cf6-b143-4fc7-91e7-69487df8809b" containerName="istio-proxy" containerID="cri-o://20d0b11ef9f5ffee8ad1c9094456d9e0654666826f5eb4d6f8d2591ba16e9241" gracePeriod=30
Oct 30 22:20:36.577750 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:36.577715    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-6bjjs"]
Oct 30 22:20:36.577989 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:36.577958    5678 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-6bjjs" podUID="275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb" containerName="istio-proxy" containerID="cri-o://275225ea6d2dcfed45a1f3ba0e0b1eb6dc329fd98ea33a7806a222b59d9aa280" gracePeriod=30
Oct 30 22:20:36.588420 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:36.588374    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9mpx4"]
Oct 30 22:20:36.588623 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:36.588596    5678 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9mpx4" podUID="103020c5-ca66-48bf-8086-1c63af040de5" containerName="istio-proxy" containerID="cri-o://90509b4a23b32df1625c3733730077caae1a914d1b7a458a659c8c59481bfe90" gracePeriod=30
Oct 30 22:20:37.051130 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:37.051093    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e0b3fa3-6bjxv"]
Oct 30 22:20:37.055199 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:37.055172    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e0b3fa3-6bjxv"]
Oct 30 22:20:38.526273 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:38.526239    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b8b45be-c9f8-4918-a934-ab66867cdb7b" path="/var/lib/kubelet/pods/0b8b45be-c9f8-4918-a934-ab66867cdb7b/volumes"
Oct 30 22:20:40.614139 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:20:40.614098    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 22:20:41.134382 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:41.133104    5678 patch_prober.go:29] interesting pod/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-f2qb5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:20:41.134382 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:41.133184    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-f2qb5" podUID="5041edee-ad23-43bf-a84e-9bb6632c2c0b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:20:41.253076 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:41.253043    5678 patch_prober.go:29] interesting pod/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-m2jqj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:20:41.253236 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:41.253107    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-m2jqj" podUID="06ae4cf6-b143-4fc7-91e7-69487df8809b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:20:41.539019 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:41.538935    5678 patch_prober.go:29] interesting pod/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9mpx4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:20:41.539019 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:41.538991    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9mpx4" podUID="103020c5-ca66-48bf-8086-1c63af040de5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:20:41.897583 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:41.897553    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-6bjjs"
Oct 30 22:20:41.929099 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:41.929069    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-f2qb5"
Oct 30 22:20:41.987062 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:41.987036    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-m2jqj"
Oct 30 22:20:41.990484 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:41.990464    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9mpx4"
Oct 30 22:20:42.031999 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.031960    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb-credential-socket\") pod \"275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb\" (UID: \"275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb\") "
Oct 30 22:20:42.032161 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.032038    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/103020c5-ca66-48bf-8086-1c63af040de5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/103020c5-ca66-48bf-8086-1c63af040de5-istio-data\") pod \"103020c5-ca66-48bf-8086-1c63af040de5\" (UID: \"103020c5-ca66-48bf-8086-1c63af040de5\") "
Oct 30 22:20:42.032161 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.032076    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5041edee-ad23-43bf-a84e-9bb6632c2c0b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5041edee-ad23-43bf-a84e-9bb6632c2c0b-workload-socket\") pod \"5041edee-ad23-43bf-a84e-9bb6632c2c0b\" (UID: \"5041edee-ad23-43bf-a84e-9bb6632c2c0b\") "
Oct 30 22:20:42.032161 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.032106    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06ae4cf6-b143-4fc7-91e7-69487df8809b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06ae4cf6-b143-4fc7-91e7-69487df8809b-workload-socket\") pod \"06ae4cf6-b143-4fc7-91e7-69487df8809b\" (UID: \"06ae4cf6-b143-4fc7-91e7-69487df8809b\") "
Oct 30 22:20:42.032161 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.032138    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5041edee-ad23-43bf-a84e-9bb6632c2c0b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5041edee-ad23-43bf-a84e-9bb6632c2c0b-istio-envoy\") pod \"5041edee-ad23-43bf-a84e-9bb6632c2c0b\" (UID: \"5041edee-ad23-43bf-a84e-9bb6632c2c0b\") "
Oct 30 22:20:42.032470 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.032184    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5041edee-ad23-43bf-a84e-9bb6632c2c0b-istio-token\" (UniqueName: \"kubernetes.io/projected/5041edee-ad23-43bf-a84e-9bb6632c2c0b-istio-token\") pod \"5041edee-ad23-43bf-a84e-9bb6632c2c0b\" (UID: \"5041edee-ad23-43bf-a84e-9bb6632c2c0b\") "
Oct 30 22:20:42.032470 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.032216    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5041edee-ad23-43bf-a84e-9bb6632c2c0b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5041edee-ad23-43bf-a84e-9bb6632c2c0b-workload-certs\") pod \"5041edee-ad23-43bf-a84e-9bb6632c2c0b\" (UID: \"5041edee-ad23-43bf-a84e-9bb6632c2c0b\") "
Oct 30 22:20:42.032470 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.032290    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb-kube-api-access-76nnv\" (UniqueName: \"kubernetes.io/projected/275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb-kube-api-access-76nnv\") pod \"275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb\" (UID: \"275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb\") "
Oct 30 22:20:42.032470 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.032327    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb-credential-socket" pod "275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb" (UID: "275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:20:42.032470 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.032370    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5041edee-ad23-43bf-a84e-9bb6632c2c0b-workload-socket" pod "5041edee-ad23-43bf-a84e-9bb6632c2c0b" (UID: "5041edee-ad23-43bf-a84e-9bb6632c2c0b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:20:42.032470 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.032373    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/103020c5-ca66-48bf-8086-1c63af040de5-istio-data" pod "103020c5-ca66-48bf-8086-1c63af040de5" (UID: "103020c5-ca66-48bf-8086-1c63af040de5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:20:42.032470 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.032377    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb-istio-podinfo\") pod \"275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb\" (UID: \"275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb\") "
Oct 30 22:20:42.032470 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.032388    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06ae4cf6-b143-4fc7-91e7-69487df8809b-workload-socket" pod "06ae4cf6-b143-4fc7-91e7-69487df8809b" (UID: "06ae4cf6-b143-4fc7-91e7-69487df8809b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:20:42.032470 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.032457    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5041edee-ad23-43bf-a84e-9bb6632c2c0b-kube-api-access-ftsb4\" (UniqueName: \"kubernetes.io/projected/5041edee-ad23-43bf-a84e-9bb6632c2c0b-kube-api-access-ftsb4\") pod \"5041edee-ad23-43bf-a84e-9bb6632c2c0b\" (UID: \"5041edee-ad23-43bf-a84e-9bb6632c2c0b\") "
Oct 30 22:20:42.032915 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.032500    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06ae4cf6-b143-4fc7-91e7-69487df8809b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06ae4cf6-b143-4fc7-91e7-69487df8809b-istiod-ca-cert\") pod \"06ae4cf6-b143-4fc7-91e7-69487df8809b\" (UID: \"06ae4cf6-b143-4fc7-91e7-69487df8809b\") "
Oct 30 22:20:42.032915 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.032545    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/103020c5-ca66-48bf-8086-1c63af040de5-istio-token\" (UniqueName: \"kubernetes.io/projected/103020c5-ca66-48bf-8086-1c63af040de5-istio-token\") pod \"103020c5-ca66-48bf-8086-1c63af040de5\" (UID: \"103020c5-ca66-48bf-8086-1c63af040de5\") "
Oct 30 22:20:42.032915 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.032652    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5041edee-ad23-43bf-a84e-9bb6632c2c0b-workload-certs" pod "5041edee-ad23-43bf-a84e-9bb6632c2c0b" (UID: "5041edee-ad23-43bf-a84e-9bb6632c2c0b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:20:42.033103 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.033082    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5041edee-ad23-43bf-a84e-9bb6632c2c0b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5041edee-ad23-43bf-a84e-9bb6632c2c0b-istio-data\") pod \"5041edee-ad23-43bf-a84e-9bb6632c2c0b\" (UID: \"5041edee-ad23-43bf-a84e-9bb6632c2c0b\") "
Oct 30 22:20:42.033160 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.033148    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06ae4cf6-b143-4fc7-91e7-69487df8809b-kube-api-access-8hqm8\" (UniqueName: \"kubernetes.io/projected/06ae4cf6-b143-4fc7-91e7-69487df8809b-kube-api-access-8hqm8\") pod \"06ae4cf6-b143-4fc7-91e7-69487df8809b\" (UID: \"06ae4cf6-b143-4fc7-91e7-69487df8809b\") "
Oct 30 22:20:42.033549 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.033520    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5041edee-ad23-43bf-a84e-9bb6632c2c0b-istio-data" pod "5041edee-ad23-43bf-a84e-9bb6632c2c0b" (UID: "5041edee-ad23-43bf-a84e-9bb6632c2c0b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:20:42.033683 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.033662    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06ae4cf6-b143-4fc7-91e7-69487df8809b-istio-token\" (UniqueName: \"kubernetes.io/projected/06ae4cf6-b143-4fc7-91e7-69487df8809b-istio-token\") pod \"06ae4cf6-b143-4fc7-91e7-69487df8809b\" (UID: \"06ae4cf6-b143-4fc7-91e7-69487df8809b\") "
Oct 30 22:20:42.033788 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.033770    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb-istio-envoy\") pod \"275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb\" (UID: \"275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb\") "
Oct 30 22:20:42.033856 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.033833    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/103020c5-ca66-48bf-8086-1c63af040de5-kube-api-access-64hrf\" (UniqueName: \"kubernetes.io/projected/103020c5-ca66-48bf-8086-1c63af040de5-kube-api-access-64hrf\") pod \"103020c5-ca66-48bf-8086-1c63af040de5\" (UID: \"103020c5-ca66-48bf-8086-1c63af040de5\") "
Oct 30 22:20:42.033917 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.033890    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb-workload-socket\") pod \"275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb\" (UID: \"275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb\") "
Oct 30 22:20:42.033980 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.033956    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/06ae4cf6-b143-4fc7-91e7-69487df8809b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06ae4cf6-b143-4fc7-91e7-69487df8809b-istio-podinfo\") pod \"06ae4cf6-b143-4fc7-91e7-69487df8809b\" (UID: \"06ae4cf6-b143-4fc7-91e7-69487df8809b\") "
Oct 30 22:20:42.034035 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.034000    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/103020c5-ca66-48bf-8086-1c63af040de5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/103020c5-ca66-48bf-8086-1c63af040de5-workload-socket\") pod \"103020c5-ca66-48bf-8086-1c63af040de5\" (UID: \"103020c5-ca66-48bf-8086-1c63af040de5\") "
Oct 30 22:20:42.034092 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.034073    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06ae4cf6-b143-4fc7-91e7-69487df8809b-istiod-ca-cert" pod "06ae4cf6-b143-4fc7-91e7-69487df8809b" (UID: "06ae4cf6-b143-4fc7-91e7-69487df8809b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:20:42.034174 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.034155    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/103020c5-ca66-48bf-8086-1c63af040de5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/103020c5-ca66-48bf-8086-1c63af040de5-istio-envoy\") pod \"103020c5-ca66-48bf-8086-1c63af040de5\" (UID: \"103020c5-ca66-48bf-8086-1c63af040de5\") "
Oct 30 22:20:42.034229 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.034195    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/103020c5-ca66-48bf-8086-1c63af040de5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/103020c5-ca66-48bf-8086-1c63af040de5-istiod-ca-cert\") pod \"103020c5-ca66-48bf-8086-1c63af040de5\" (UID: \"103020c5-ca66-48bf-8086-1c63af040de5\") "
Oct 30 22:20:42.034281 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.034245    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06ae4cf6-b143-4fc7-91e7-69487df8809b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06ae4cf6-b143-4fc7-91e7-69487df8809b-istio-data\") pod \"06ae4cf6-b143-4fc7-91e7-69487df8809b\" (UID: \"06ae4cf6-b143-4fc7-91e7-69487df8809b\") "
Oct 30 22:20:42.034577 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.034293    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/103020c5-ca66-48bf-8086-1c63af040de5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/103020c5-ca66-48bf-8086-1c63af040de5-workload-certs\") pod \"103020c5-ca66-48bf-8086-1c63af040de5\" (UID: \"103020c5-ca66-48bf-8086-1c63af040de5\") "
Oct 30 22:20:42.034577 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.034357    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/103020c5-ca66-48bf-8086-1c63af040de5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/103020c5-ca66-48bf-8086-1c63af040de5-istio-podinfo\") pod \"103020c5-ca66-48bf-8086-1c63af040de5\" (UID: \"103020c5-ca66-48bf-8086-1c63af040de5\") "
Oct 30 22:20:42.034577 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.034424    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/103020c5-ca66-48bf-8086-1c63af040de5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/103020c5-ca66-48bf-8086-1c63af040de5-credential-socket\") pod \"103020c5-ca66-48bf-8086-1c63af040de5\" (UID: \"103020c5-ca66-48bf-8086-1c63af040de5\") "
Oct 30 22:20:42.034577 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.034460    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5041edee-ad23-43bf-a84e-9bb6632c2c0b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5041edee-ad23-43bf-a84e-9bb6632c2c0b-credential-socket\") pod \"5041edee-ad23-43bf-a84e-9bb6632c2c0b\" (UID: \"5041edee-ad23-43bf-a84e-9bb6632c2c0b\") "
Oct 30 22:20:42.034577 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.034535    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06ae4cf6-b143-4fc7-91e7-69487df8809b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06ae4cf6-b143-4fc7-91e7-69487df8809b-credential-socket\") pod \"06ae4cf6-b143-4fc7-91e7-69487df8809b\" (UID: \"06ae4cf6-b143-4fc7-91e7-69487df8809b\") "
Oct 30 22:20:42.034577 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.034566    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06ae4cf6-b143-4fc7-91e7-69487df8809b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06ae4cf6-b143-4fc7-91e7-69487df8809b-istio-envoy\") pod \"06ae4cf6-b143-4fc7-91e7-69487df8809b\" (UID: \"06ae4cf6-b143-4fc7-91e7-69487df8809b\") "
Oct 30 22:20:42.034918 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.034618    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb-istiod-ca-cert\") pod \"275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb\" (UID: \"275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb\") "
Oct 30 22:20:42.034918 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.034656    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06ae4cf6-b143-4fc7-91e7-69487df8809b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06ae4cf6-b143-4fc7-91e7-69487df8809b-workload-certs\") pod \"06ae4cf6-b143-4fc7-91e7-69487df8809b\" (UID: \"06ae4cf6-b143-4fc7-91e7-69487df8809b\") "
Oct 30 22:20:42.034918 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.034719    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb-istio-data\") pod \"275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb\" (UID: \"275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb\") "
Oct 30 22:20:42.034918 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.034804    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5041edee-ad23-43bf-a84e-9bb6632c2c0b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5041edee-ad23-43bf-a84e-9bb6632c2c0b-istiod-ca-cert\") pod \"5041edee-ad23-43bf-a84e-9bb6632c2c0b\" (UID: \"5041edee-ad23-43bf-a84e-9bb6632c2c0b\") "
Oct 30 22:20:42.034918 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.034851    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb-istio-token\" (UniqueName: \"kubernetes.io/projected/275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb-istio-token\") pod \"275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb\" (UID: \"275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb\") "
Oct 30 22:20:42.035154 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.034922    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb-workload-certs\") pod \"275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb\" (UID: \"275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb\") "
Oct 30 22:20:42.035154 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.034972    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5041edee-ad23-43bf-a84e-9bb6632c2c0b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5041edee-ad23-43bf-a84e-9bb6632c2c0b-istio-podinfo\") pod \"5041edee-ad23-43bf-a84e-9bb6632c2c0b\" (UID: \"5041edee-ad23-43bf-a84e-9bb6632c2c0b\") "
Oct 30 22:20:42.037670 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.037362    5678 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb-credential-socket\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:20:42.037670 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.037391    5678 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/103020c5-ca66-48bf-8086-1c63af040de5-istio-data\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:20:42.037670 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.037422    5678 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5041edee-ad23-43bf-a84e-9bb6632c2c0b-workload-socket\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:20:42.037670 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.037447    5678 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06ae4cf6-b143-4fc7-91e7-69487df8809b-workload-socket\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:20:42.037670 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.037464    5678 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5041edee-ad23-43bf-a84e-9bb6632c2c0b-workload-certs\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:20:42.037670 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.037481    5678 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06ae4cf6-b143-4fc7-91e7-69487df8809b-istiod-ca-cert\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:20:42.037670 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.037496    5678 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5041edee-ad23-43bf-a84e-9bb6632c2c0b-istio-data\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:20:42.038116 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.038083    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb-kube-api-access-76nnv" pod "275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb" (UID: "275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb"). InnerVolumeSpecName "kube-api-access-76nnv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:20:42.039935 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.038208    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb-workload-socket" pod "275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb" (UID: "275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:20:42.039935 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.038226    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5041edee-ad23-43bf-a84e-9bb6632c2c0b-kube-api-access-ftsb4" pod "5041edee-ad23-43bf-a84e-9bb6632c2c0b" (UID: "5041edee-ad23-43bf-a84e-9bb6632c2c0b"). InnerVolumeSpecName "kube-api-access-ftsb4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:20:42.039935 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.038308    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06ae4cf6-b143-4fc7-91e7-69487df8809b-kube-api-access-8hqm8" pod "06ae4cf6-b143-4fc7-91e7-69487df8809b" (UID: "06ae4cf6-b143-4fc7-91e7-69487df8809b"). InnerVolumeSpecName "kube-api-access-8hqm8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:20:42.039935 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.038523    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06ae4cf6-b143-4fc7-91e7-69487df8809b-workload-certs" pod "06ae4cf6-b143-4fc7-91e7-69487df8809b" (UID: "06ae4cf6-b143-4fc7-91e7-69487df8809b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:20:42.039935 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.038611    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06ae4cf6-b143-4fc7-91e7-69487df8809b-credential-socket" pod "06ae4cf6-b143-4fc7-91e7-69487df8809b" (UID: "06ae4cf6-b143-4fc7-91e7-69487df8809b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:20:42.039935 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.038896    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb-istiod-ca-cert" pod "275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb" (UID: "275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:20:42.039935 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.039072    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb-istio-data" pod "275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb" (UID: "275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:20:42.039935 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.039309    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/103020c5-ca66-48bf-8086-1c63af040de5-workload-certs" pod "103020c5-ca66-48bf-8086-1c63af040de5" (UID: "103020c5-ca66-48bf-8086-1c63af040de5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:20:42.039935 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.039349    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06ae4cf6-b143-4fc7-91e7-69487df8809b-istio-data" pod "06ae4cf6-b143-4fc7-91e7-69487df8809b" (UID: "06ae4cf6-b143-4fc7-91e7-69487df8809b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:20:42.039935 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.039535    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/103020c5-ca66-48bf-8086-1c63af040de5-credential-socket" pod "103020c5-ca66-48bf-8086-1c63af040de5" (UID: "103020c5-ca66-48bf-8086-1c63af040de5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:20:42.039935 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.039545    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/103020c5-ca66-48bf-8086-1c63af040de5-workload-socket" pod "103020c5-ca66-48bf-8086-1c63af040de5" (UID: "103020c5-ca66-48bf-8086-1c63af040de5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:20:42.039935 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.039566    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5041edee-ad23-43bf-a84e-9bb6632c2c0b-credential-socket" pod "5041edee-ad23-43bf-a84e-9bb6632c2c0b" (UID: "5041edee-ad23-43bf-a84e-9bb6632c2c0b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:20:42.039935 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.039783    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb-workload-certs" pod "275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb" (UID: "275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:20:42.040655 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.039893    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5041edee-ad23-43bf-a84e-9bb6632c2c0b-istiod-ca-cert" pod "5041edee-ad23-43bf-a84e-9bb6632c2c0b" (UID: "5041edee-ad23-43bf-a84e-9bb6632c2c0b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:20:42.043801 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.043773    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/103020c5-ca66-48bf-8086-1c63af040de5-istiod-ca-cert" pod "103020c5-ca66-48bf-8086-1c63af040de5" (UID: "103020c5-ca66-48bf-8086-1c63af040de5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:20:42.050339 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.049804    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb-istio-podinfo" pod "275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb" (UID: "275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:20:42.050339 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.049820    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5041edee-ad23-43bf-a84e-9bb6632c2c0b-istio-podinfo" pod "5041edee-ad23-43bf-a84e-9bb6632c2c0b" (UID: "5041edee-ad23-43bf-a84e-9bb6632c2c0b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:20:42.050339 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.049826    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5041edee-ad23-43bf-a84e-9bb6632c2c0b-istio-envoy" pod "5041edee-ad23-43bf-a84e-9bb6632c2c0b" (UID: "5041edee-ad23-43bf-a84e-9bb6632c2c0b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:20:42.050339 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.049843    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/103020c5-ca66-48bf-8086-1c63af040de5-istio-podinfo" pod "103020c5-ca66-48bf-8086-1c63af040de5" (UID: "103020c5-ca66-48bf-8086-1c63af040de5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:20:42.050339 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.049872    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/06ae4cf6-b143-4fc7-91e7-69487df8809b-istio-podinfo" pod "06ae4cf6-b143-4fc7-91e7-69487df8809b" (UID: "06ae4cf6-b143-4fc7-91e7-69487df8809b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:20:42.050339 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.050051    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06ae4cf6-b143-4fc7-91e7-69487df8809b-istio-token" pod "06ae4cf6-b143-4fc7-91e7-69487df8809b" (UID: "06ae4cf6-b143-4fc7-91e7-69487df8809b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:20:42.050339 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.050325    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06ae4cf6-b143-4fc7-91e7-69487df8809b-istio-envoy" pod "06ae4cf6-b143-4fc7-91e7-69487df8809b" (UID: "06ae4cf6-b143-4fc7-91e7-69487df8809b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:20:42.050714 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.050380    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5041edee-ad23-43bf-a84e-9bb6632c2c0b-istio-token" pod "5041edee-ad23-43bf-a84e-9bb6632c2c0b" (UID: "5041edee-ad23-43bf-a84e-9bb6632c2c0b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:20:42.050714 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.050420    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/103020c5-ca66-48bf-8086-1c63af040de5-istio-envoy" pod "103020c5-ca66-48bf-8086-1c63af040de5" (UID: "103020c5-ca66-48bf-8086-1c63af040de5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:20:42.050714 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.050475    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb-istio-token" pod "275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb" (UID: "275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:20:42.050812 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.050769    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/103020c5-ca66-48bf-8086-1c63af040de5-kube-api-access-64hrf" pod "103020c5-ca66-48bf-8086-1c63af040de5" (UID: "103020c5-ca66-48bf-8086-1c63af040de5"). InnerVolumeSpecName "kube-api-access-64hrf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:20:42.051169 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.051140    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb-istio-envoy" pod "275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb" (UID: "275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:20:42.051535 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.051516    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/103020c5-ca66-48bf-8086-1c63af040de5-istio-token" pod "103020c5-ca66-48bf-8086-1c63af040de5" (UID: "103020c5-ca66-48bf-8086-1c63af040de5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:20:42.128240 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.128205    5678 generic.go:356] "Generic (PLEG): container finished" podID="5041edee-ad23-43bf-a84e-9bb6632c2c0b" containerID="25ad9f97a8fe5ee3c92c23e1e4730013643b77854b77e606209453c96d9cd173" exitCode=0
Oct 30 22:20:42.128475 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.128286    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-f2qb5"
Oct 30 22:20:42.128475 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.128300    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-f2qb5" event={"ID":"5041edee-ad23-43bf-a84e-9bb6632c2c0b","Type":"ContainerDied","Data":"25ad9f97a8fe5ee3c92c23e1e4730013643b77854b77e606209453c96d9cd173"}
Oct 30 22:20:42.128475 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.128436    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-f2qb5" event={"ID":"5041edee-ad23-43bf-a84e-9bb6632c2c0b","Type":"ContainerDied","Data":"e4e58202088d7da9ecf9394102bf55a90e501e65e36836b567a63a9387b8ed51"}
Oct 30 22:20:42.128475 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.128475    5678 scope.go:122] "RemoveContainer" containerID="25ad9f97a8fe5ee3c92c23e1e4730013643b77854b77e606209453c96d9cd173"
Oct 30 22:20:42.130563 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.130539    5678 generic.go:356] "Generic (PLEG): container finished" podID="103020c5-ca66-48bf-8086-1c63af040de5" containerID="90509b4a23b32df1625c3733730077caae1a914d1b7a458a659c8c59481bfe90" exitCode=0
Oct 30 22:20:42.130681 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.130610    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9mpx4"
Oct 30 22:20:42.130681 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.130619    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9mpx4" event={"ID":"103020c5-ca66-48bf-8086-1c63af040de5","Type":"ContainerDied","Data":"90509b4a23b32df1625c3733730077caae1a914d1b7a458a659c8c59481bfe90"}
Oct 30 22:20:42.130799 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.130690    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9mpx4" event={"ID":"103020c5-ca66-48bf-8086-1c63af040de5","Type":"ContainerDied","Data":"ebccdb4490e2ffbb7c440fdad951ceac418b9ee654bf2574aa088501baed65f3"}
Oct 30 22:20:42.132356 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.132331    5678 generic.go:356] "Generic (PLEG): container finished" podID="06ae4cf6-b143-4fc7-91e7-69487df8809b" containerID="20d0b11ef9f5ffee8ad1c9094456d9e0654666826f5eb4d6f8d2591ba16e9241" exitCode=0
Oct 30 22:20:42.132488 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.132429    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-m2jqj" event={"ID":"06ae4cf6-b143-4fc7-91e7-69487df8809b","Type":"ContainerDied","Data":"20d0b11ef9f5ffee8ad1c9094456d9e0654666826f5eb4d6f8d2591ba16e9241"}
Oct 30 22:20:42.132488 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.132461    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-m2jqj"
Oct 30 22:20:42.132488 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.132476    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-m2jqj" event={"ID":"06ae4cf6-b143-4fc7-91e7-69487df8809b","Type":"ContainerDied","Data":"9316adb8049175d98105879f22d894a734d0880613c0bc76c986f8861d83c3d4"}
Oct 30 22:20:42.134375 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.134349    5678 generic.go:356] "Generic (PLEG): container finished" podID="275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb" containerID="275225ea6d2dcfed45a1f3ba0e0b1eb6dc329fd98ea33a7806a222b59d9aa280" exitCode=0
Oct 30 22:20:42.134560 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.134538    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-6bjjs" event={"ID":"275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb","Type":"ContainerDied","Data":"275225ea6d2dcfed45a1f3ba0e0b1eb6dc329fd98ea33a7806a222b59d9aa280"}
Oct 30 22:20:42.134625 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.134587    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-6bjjs" event={"ID":"275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb","Type":"ContainerDied","Data":"c132fa00e8c358c42c15e7c8266725394f4b9e86c6ed3f5c185317865911e6b8"}
Oct 30 22:20:42.134686 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.134672    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-6bjjs"
Oct 30 22:20:42.138797 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.138510    5678 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb-workload-socket\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:20:42.138797 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.138534    5678 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06ae4cf6-b143-4fc7-91e7-69487df8809b-istio-podinfo\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:20:42.138797 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.138551    5678 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/103020c5-ca66-48bf-8086-1c63af040de5-workload-socket\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:20:42.138797 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.138566    5678 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/103020c5-ca66-48bf-8086-1c63af040de5-istio-envoy\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:20:42.138797 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.138580    5678 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/103020c5-ca66-48bf-8086-1c63af040de5-istiod-ca-cert\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:20:42.138797 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.138593    5678 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06ae4cf6-b143-4fc7-91e7-69487df8809b-istio-data\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:20:42.138797 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.138606    5678 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/103020c5-ca66-48bf-8086-1c63af040de5-workload-certs\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:20:42.138797 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.138620    5678 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/103020c5-ca66-48bf-8086-1c63af040de5-istio-podinfo\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:20:42.138797 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.138661    5678 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/103020c5-ca66-48bf-8086-1c63af040de5-credential-socket\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:20:42.138797 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.138675    5678 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5041edee-ad23-43bf-a84e-9bb6632c2c0b-credential-socket\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:20:42.138797 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.138724    5678 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06ae4cf6-b143-4fc7-91e7-69487df8809b-credential-socket\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:20:42.138797 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.138739    5678 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06ae4cf6-b143-4fc7-91e7-69487df8809b-istio-envoy\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:20:42.138797 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.138748    5678 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb-istiod-ca-cert\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:20:42.138797 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.138758    5678 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06ae4cf6-b143-4fc7-91e7-69487df8809b-workload-certs\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:20:42.138797 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.138767    5678 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb-istio-data\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:20:42.138797 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.138778    5678 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5041edee-ad23-43bf-a84e-9bb6632c2c0b-istiod-ca-cert\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:20:42.138797 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.138788    5678 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb-istio-token\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:20:42.138797 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.138801    5678 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb-workload-certs\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:20:42.138797 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.138810    5678 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5041edee-ad23-43bf-a84e-9bb6632c2c0b-istio-podinfo\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:20:42.139852 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.138820    5678 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5041edee-ad23-43bf-a84e-9bb6632c2c0b-istio-envoy\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:20:42.139852 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.138828    5678 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5041edee-ad23-43bf-a84e-9bb6632c2c0b-istio-token\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:20:42.139852 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.138837    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76nnv\" (UniqueName: \"kubernetes.io/projected/275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb-kube-api-access-76nnv\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:20:42.139852 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.138850    5678 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb-istio-podinfo\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:20:42.139852 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.138864    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftsb4\" (UniqueName: \"kubernetes.io/projected/5041edee-ad23-43bf-a84e-9bb6632c2c0b-kube-api-access-ftsb4\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:20:42.139852 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.138876    5678 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/103020c5-ca66-48bf-8086-1c63af040de5-istio-token\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:20:42.139852 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.138889    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hqm8\" (UniqueName: \"kubernetes.io/projected/06ae4cf6-b143-4fc7-91e7-69487df8809b-kube-api-access-8hqm8\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:20:42.139852 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.138902    5678 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06ae4cf6-b143-4fc7-91e7-69487df8809b-istio-token\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:20:42.139852 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.138914    5678 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb-istio-envoy\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:20:42.139852 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.138928    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64hrf\" (UniqueName: \"kubernetes.io/projected/103020c5-ca66-48bf-8086-1c63af040de5-kube-api-access-64hrf\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:20:42.149596 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.145384    5678 scope.go:122] "RemoveContainer" containerID="25ad9f97a8fe5ee3c92c23e1e4730013643b77854b77e606209453c96d9cd173"
Oct 30 22:20:42.149596 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:20:42.145698    5678 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25ad9f97a8fe5ee3c92c23e1e4730013643b77854b77e606209453c96d9cd173\": container with ID starting with 25ad9f97a8fe5ee3c92c23e1e4730013643b77854b77e606209453c96d9cd173 not found: ID does not exist" containerID="25ad9f97a8fe5ee3c92c23e1e4730013643b77854b77e606209453c96d9cd173"
Oct 30 22:20:42.149596 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.145722    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25ad9f97a8fe5ee3c92c23e1e4730013643b77854b77e606209453c96d9cd173"} err="failed to get container status \"25ad9f97a8fe5ee3c92c23e1e4730013643b77854b77e606209453c96d9cd173\": rpc error: code = NotFound desc = could not find container \"25ad9f97a8fe5ee3c92c23e1e4730013643b77854b77e606209453c96d9cd173\": container with ID starting with 25ad9f97a8fe5ee3c92c23e1e4730013643b77854b77e606209453c96d9cd173 not found: ID does not exist"
Oct 30 22:20:42.149596 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.145739    5678 scope.go:122] "RemoveContainer" containerID="90509b4a23b32df1625c3733730077caae1a914d1b7a458a659c8c59481bfe90"
Oct 30 22:20:42.160010 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.159989    5678 scope.go:122] "RemoveContainer" containerID="90509b4a23b32df1625c3733730077caae1a914d1b7a458a659c8c59481bfe90"
Oct 30 22:20:42.160441 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:20:42.160420    5678 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90509b4a23b32df1625c3733730077caae1a914d1b7a458a659c8c59481bfe90\": container with ID starting with 90509b4a23b32df1625c3733730077caae1a914d1b7a458a659c8c59481bfe90 not found: ID does not exist" containerID="90509b4a23b32df1625c3733730077caae1a914d1b7a458a659c8c59481bfe90"
Oct 30 22:20:42.160550 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.160445    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90509b4a23b32df1625c3733730077caae1a914d1b7a458a659c8c59481bfe90"} err="failed to get container status \"90509b4a23b32df1625c3733730077caae1a914d1b7a458a659c8c59481bfe90\": rpc error: code = NotFound desc = could not find container \"90509b4a23b32df1625c3733730077caae1a914d1b7a458a659c8c59481bfe90\": container with ID starting with 90509b4a23b32df1625c3733730077caae1a914d1b7a458a659c8c59481bfe90 not found: ID does not exist"
Oct 30 22:20:42.160550 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.160460    5678 scope.go:122] "RemoveContainer" containerID="20d0b11ef9f5ffee8ad1c9094456d9e0654666826f5eb4d6f8d2591ba16e9241"
Oct 30 22:20:42.171790 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.171775    5678 scope.go:122] "RemoveContainer" containerID="20d0b11ef9f5ffee8ad1c9094456d9e0654666826f5eb4d6f8d2591ba16e9241"
Oct 30 22:20:42.172085 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:20:42.172066    5678 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20d0b11ef9f5ffee8ad1c9094456d9e0654666826f5eb4d6f8d2591ba16e9241\": container with ID starting with 20d0b11ef9f5ffee8ad1c9094456d9e0654666826f5eb4d6f8d2591ba16e9241 not found: ID does not exist" containerID="20d0b11ef9f5ffee8ad1c9094456d9e0654666826f5eb4d6f8d2591ba16e9241"
Oct 30 22:20:42.172149 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.172090    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20d0b11ef9f5ffee8ad1c9094456d9e0654666826f5eb4d6f8d2591ba16e9241"} err="failed to get container status \"20d0b11ef9f5ffee8ad1c9094456d9e0654666826f5eb4d6f8d2591ba16e9241\": rpc error: code = NotFound desc = could not find container \"20d0b11ef9f5ffee8ad1c9094456d9e0654666826f5eb4d6f8d2591ba16e9241\": container with ID starting with 20d0b11ef9f5ffee8ad1c9094456d9e0654666826f5eb4d6f8d2591ba16e9241 not found: ID does not exist"
Oct 30 22:20:42.172149 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.172114    5678 scope.go:122] "RemoveContainer" containerID="275225ea6d2dcfed45a1f3ba0e0b1eb6dc329fd98ea33a7806a222b59d9aa280"
Oct 30 22:20:42.182234 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.182216    5678 scope.go:122] "RemoveContainer" containerID="275225ea6d2dcfed45a1f3ba0e0b1eb6dc329fd98ea33a7806a222b59d9aa280"
Oct 30 22:20:42.182503 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:20:42.182485    5678 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"275225ea6d2dcfed45a1f3ba0e0b1eb6dc329fd98ea33a7806a222b59d9aa280\": container with ID starting with 275225ea6d2dcfed45a1f3ba0e0b1eb6dc329fd98ea33a7806a222b59d9aa280 not found: ID does not exist" containerID="275225ea6d2dcfed45a1f3ba0e0b1eb6dc329fd98ea33a7806a222b59d9aa280"
Oct 30 22:20:42.182571 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.182508    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"275225ea6d2dcfed45a1f3ba0e0b1eb6dc329fd98ea33a7806a222b59d9aa280"} err="failed to get container status \"275225ea6d2dcfed45a1f3ba0e0b1eb6dc329fd98ea33a7806a222b59d9aa280\": rpc error: code = NotFound desc = could not find container \"275225ea6d2dcfed45a1f3ba0e0b1eb6dc329fd98ea33a7806a222b59d9aa280\": container with ID starting with 275225ea6d2dcfed45a1f3ba0e0b1eb6dc329fd98ea33a7806a222b59d9aa280 not found: ID does not exist"
Oct 30 22:20:42.535031 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.534748    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9mpx4"]
Oct 30 22:20:42.539316 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.539292    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9mpx4"]
Oct 30 22:20:42.673024 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.672982    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-m2jqj"]
Oct 30 22:20:42.690876 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.690846    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-m2jqj"]
Oct 30 22:20:42.829226 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.829140    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-f2qb5"]
Oct 30 22:20:42.862424 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.862383    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-f2qb5"]
Oct 30 22:20:42.969588 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.969557    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-6bjjs"]
Oct 30 22:20:42.978919 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:42.978633    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-6bjjs"]
Oct 30 22:20:43.058096 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:43.058068    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72eaded0-sqfvh"]
Oct 30 22:20:43.061893 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:43.061869    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72eaded0-sqfvh"]
Oct 30 22:20:44.525296 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:44.525266    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06ae4cf6-b143-4fc7-91e7-69487df8809b" path="/var/lib/kubelet/pods/06ae4cf6-b143-4fc7-91e7-69487df8809b/volumes"
Oct 30 22:20:44.525865 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:44.525847    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="103020c5-ca66-48bf-8086-1c63af040de5" path="/var/lib/kubelet/pods/103020c5-ca66-48bf-8086-1c63af040de5/volumes"
Oct 30 22:20:44.526258 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:44.526245    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f1f5fbe-f541-45aa-89b1-a0971f43c2cf" path="/var/lib/kubelet/pods/1f1f5fbe-f541-45aa-89b1-a0971f43c2cf/volumes"
Oct 30 22:20:44.526857 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:44.526838    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb" path="/var/lib/kubelet/pods/275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb/volumes"
Oct 30 22:20:44.527277 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:44.527263    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5041edee-ad23-43bf-a84e-9bb6632c2c0b" path="/var/lib/kubelet/pods/5041edee-ad23-43bf-a84e-9bb6632c2c0b/volumes"
Oct 30 22:20:59.426956 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:59.426923    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a98f5ef3-rtq8s"]
Oct 30 22:20:59.427585 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:59.427571    5678 cpu_manager.go:414] "containerMap: removing container" podUID="06ae4cf6-b143-4fc7-91e7-69487df8809b" containerName="istio-proxy"
Oct 30 22:20:59.427648 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:59.427587    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06ae4cf6-b143-4fc7-91e7-69487df8809b" containerName="istio-proxy"
Oct 30 22:20:59.427648 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:59.427620    5678 cpu_manager.go:414] "containerMap: removing container" podUID="669af24d-752d-407e-af9b-8ba5c02c40de" containerName="istio-proxy"
Oct 30 22:20:59.427648 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:59.427626    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="669af24d-752d-407e-af9b-8ba5c02c40de" containerName="istio-proxy"
Oct 30 22:20:59.427648 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:59.427642    5678 cpu_manager.go:414] "containerMap: removing container" podUID="5041edee-ad23-43bf-a84e-9bb6632c2c0b" containerName="istio-proxy"
Oct 30 22:20:59.427648 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:59.427647    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5041edee-ad23-43bf-a84e-9bb6632c2c0b" containerName="istio-proxy"
Oct 30 22:20:59.427805 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:59.427655    5678 cpu_manager.go:414] "containerMap: removing container" podUID="103020c5-ca66-48bf-8086-1c63af040de5" containerName="istio-proxy"
Oct 30 22:20:59.427805 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:59.427659    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="103020c5-ca66-48bf-8086-1c63af040de5" containerName="istio-proxy"
Oct 30 22:20:59.427805 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:59.427668    5678 cpu_manager.go:414] "containerMap: removing container" podUID="275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb" containerName="istio-proxy"
Oct 30 22:20:59.427805 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:59.427673    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb" containerName="istio-proxy"
Oct 30 22:20:59.427805 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:59.427689    5678 cpu_manager.go:414] "containerMap: removing container" podUID="ef450e67-6b32-4617-a73d-c3d5ccbbf203" containerName="payload-pre-processing"
Oct 30 22:20:59.427805 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:59.427695    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef450e67-6b32-4617-a73d-c3d5ccbbf203" containerName="payload-pre-processing"
Oct 30 22:20:59.427805 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:59.427777    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="669af24d-752d-407e-af9b-8ba5c02c40de" containerName="istio-proxy"
Oct 30 22:20:59.427805 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:59.427786    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="5041edee-ad23-43bf-a84e-9bb6632c2c0b" containerName="istio-proxy"
Oct 30 22:20:59.427805 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:59.427793    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="103020c5-ca66-48bf-8086-1c63af040de5" containerName="istio-proxy"
Oct 30 22:20:59.427805 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:59.427798    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="275c9ec2-0c92-4e50-b6b7-a1f82c02d3eb" containerName="istio-proxy"
Oct 30 22:20:59.427805 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:59.427806    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef450e67-6b32-4617-a73d-c3d5ccbbf203" containerName="payload-pre-processing"
Oct 30 22:20:59.428326 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:59.427813    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="06ae4cf6-b143-4fc7-91e7-69487df8809b" containerName="istio-proxy"
Oct 30 22:20:59.431942 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:59.431924    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a98f5ef3-rtq8s"
Oct 30 22:20:59.486795 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:59.486696    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a98f5ef3-rtq8s"]
Oct 30 22:20:59.528350 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:59.528313    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:20:59.528350 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:59.528345    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5rvcj\""
Oct 30 22:20:59.538516 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:59.538489    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/defca364-7e29-483a-b948-d41e8bb31cd9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a98f5ef3-rtq8s\" (UID: \"defca364-7e29-483a-b948-d41e8bb31cd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a98f5ef3-rtq8s"
Oct 30 22:20:59.538637 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:59.538600    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/defca364-7e29-483a-b948-d41e8bb31cd9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a98f5ef3-rtq8s\" (UID: \"defca364-7e29-483a-b948-d41e8bb31cd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a98f5ef3-rtq8s"
Oct 30 22:20:59.538823 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:59.538804    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cx5b6\" (UniqueName: \"kubernetes.io/projected/defca364-7e29-483a-b948-d41e8bb31cd9-kube-api-access-cx5b6\") pod \"maas-api-revoke-sub-a98f5ef3-rtq8s\" (UID: \"defca364-7e29-483a-b948-d41e8bb31cd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a98f5ef3-rtq8s"
Oct 30 22:20:59.640141 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:59.640103    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/defca364-7e29-483a-b948-d41e8bb31cd9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a98f5ef3-rtq8s\" (UID: \"defca364-7e29-483a-b948-d41e8bb31cd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a98f5ef3-rtq8s"
Oct 30 22:20:59.640337 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:59.640320    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cx5b6\" (UniqueName: \"kubernetes.io/projected/defca364-7e29-483a-b948-d41e8bb31cd9-kube-api-access-cx5b6\") pod \"maas-api-revoke-sub-a98f5ef3-rtq8s\" (UID: \"defca364-7e29-483a-b948-d41e8bb31cd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a98f5ef3-rtq8s"
Oct 30 22:20:59.640486 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:59.640471    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/defca364-7e29-483a-b948-d41e8bb31cd9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a98f5ef3-rtq8s\" (UID: \"defca364-7e29-483a-b948-d41e8bb31cd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a98f5ef3-rtq8s"
Oct 30 22:20:59.640844 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:59.640824    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/defca364-7e29-483a-b948-d41e8bb31cd9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a98f5ef3-rtq8s\" (UID: \"defca364-7e29-483a-b948-d41e8bb31cd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a98f5ef3-rtq8s"
Oct 30 22:20:59.642704 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:59.642687    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/defca364-7e29-483a-b948-d41e8bb31cd9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a98f5ef3-rtq8s\" (UID: \"defca364-7e29-483a-b948-d41e8bb31cd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a98f5ef3-rtq8s"
Oct 30 22:20:59.699354 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:59.699269    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cx5b6\" (UniqueName: \"kubernetes.io/projected/defca364-7e29-483a-b948-d41e8bb31cd9-kube-api-access-cx5b6\") pod \"maas-api-revoke-sub-a98f5ef3-rtq8s\" (UID: \"defca364-7e29-483a-b948-d41e8bb31cd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a98f5ef3-rtq8s"
Oct 30 22:20:59.744974 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:20:59.744939    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a98f5ef3-rtq8s"
Oct 30 22:21:00.139286 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:21:00.139258    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddefca364_7e29_483a_b948_d41e8bb31cd9.slice/crio-1c67bbf1261b15818781198659f9e2d176b50190af9053f63a73f846870968e5 WatchSource:0}: Error finding container 1c67bbf1261b15818781198659f9e2d176b50190af9053f63a73f846870968e5: Status 404 returned error can't find the container with id 1c67bbf1261b15818781198659f9e2d176b50190af9053f63a73f846870968e5
Oct 30 22:21:00.140363 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:00.140335    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a98f5ef3-rtq8s"]
Oct 30 22:21:00.279025 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:00.278994    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a98f5ef3-rtq8s" event={"ID":"defca364-7e29-483a-b948-d41e8bb31cd9","Type":"ContainerStarted","Data":"1c67bbf1261b15818781198659f9e2d176b50190af9053f63a73f846870968e5"}
Oct 30 22:21:01.291829 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:01.291700    5678 generic.go:356] "Generic (PLEG): container finished" podID="defca364-7e29-483a-b948-d41e8bb31cd9" containerID="2a09d6934cb79833e0a16eba32df437c3f53da4db96843f212129ad58aa81a7b" exitCode=0
Oct 30 22:21:01.292285 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:01.291961    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a98f5ef3-rtq8s" event={"ID":"defca364-7e29-483a-b948-d41e8bb31cd9","Type":"ContainerDied","Data":"2a09d6934cb79833e0a16eba32df437c3f53da4db96843f212129ad58aa81a7b"}
Oct 30 22:21:02.459582 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:02.459558    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a98f5ef3-rtq8s"
Oct 30 22:21:02.583448 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:02.583364    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/defca364-7e29-483a-b948-d41e8bb31cd9-kube-api-access-cx5b6\" (UniqueName: \"kubernetes.io/projected/defca364-7e29-483a-b948-d41e8bb31cd9-kube-api-access-cx5b6\") pod \"defca364-7e29-483a-b948-d41e8bb31cd9\" (UID: \"defca364-7e29-483a-b948-d41e8bb31cd9\") "
Oct 30 22:21:02.583607 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:02.583461    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/defca364-7e29-483a-b948-d41e8bb31cd9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/defca364-7e29-483a-b948-d41e8bb31cd9-maas-api-xks-ca\") pod \"defca364-7e29-483a-b948-d41e8bb31cd9\" (UID: \"defca364-7e29-483a-b948-d41e8bb31cd9\") "
Oct 30 22:21:02.583607 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:02.583589    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/defca364-7e29-483a-b948-d41e8bb31cd9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/defca364-7e29-483a-b948-d41e8bb31cd9-maas-api-openshift-ca\") pod \"defca364-7e29-483a-b948-d41e8bb31cd9\" (UID: \"defca364-7e29-483a-b948-d41e8bb31cd9\") "
Oct 30 22:21:02.584006 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:02.583978    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/defca364-7e29-483a-b948-d41e8bb31cd9-maas-api-openshift-ca" pod "defca364-7e29-483a-b948-d41e8bb31cd9" (UID: "defca364-7e29-483a-b948-d41e8bb31cd9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:21:02.584336 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:02.584311    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/defca364-7e29-483a-b948-d41e8bb31cd9-maas-api-openshift-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:21:02.586049 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:02.586020    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/defca364-7e29-483a-b948-d41e8bb31cd9-kube-api-access-cx5b6" pod "defca364-7e29-483a-b948-d41e8bb31cd9" (UID: "defca364-7e29-483a-b948-d41e8bb31cd9"). InnerVolumeSpecName "kube-api-access-cx5b6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:21:02.586541 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:02.586509    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/defca364-7e29-483a-b948-d41e8bb31cd9-maas-api-xks-ca" pod "defca364-7e29-483a-b948-d41e8bb31cd9" (UID: "defca364-7e29-483a-b948-d41e8bb31cd9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:21:02.685344 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:02.685311    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cx5b6\" (UniqueName: \"kubernetes.io/projected/defca364-7e29-483a-b948-d41e8bb31cd9-kube-api-access-cx5b6\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:21:02.685344 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:02.685340    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/defca364-7e29-483a-b948-d41e8bb31cd9-maas-api-xks-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:21:03.307610 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:03.307582    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a98f5ef3-rtq8s"
Oct 30 22:21:03.307766 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:03.307613    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a98f5ef3-rtq8s" event={"ID":"defca364-7e29-483a-b948-d41e8bb31cd9","Type":"ContainerDied","Data":"1c67bbf1261b15818781198659f9e2d176b50190af9053f63a73f846870968e5"}
Oct 30 22:21:03.307766 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:03.307662    5678 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1c67bbf1261b15818781198659f9e2d176b50190af9053f63a73f846870968e5"
Oct 30 22:21:10.579434 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:21:10.578546    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 22:21:23.388891 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:23.388858    5678 scope.go:122] "RemoveContainer" containerID="8307e52fa9d117041c02f3569f43909a040daa38fa411cd9f4cb4b648c66a629"
Oct 30 22:21:23.403378 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:23.403349    5678 scope.go:122] "RemoveContainer" containerID="b2baf0fdf7d7d08e4267d9745aaa28b8623dc6d647d4dfa89f24cc0b0af05fbc"
Oct 30 22:21:23.413081 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:23.413059    5678 scope.go:122] "RemoveContainer" containerID="b7f33affc7b6076c3dded4f89c2804eca32fc64323012fcc55cd108e879a23ac"
Oct 30 22:21:23.422572 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:23.422552    5678 scope.go:122] "RemoveContainer" containerID="d3302a639532d27ee202582e86eef078337558fab6447c1bc794c0e7ab66bce0"
Oct 30 22:21:23.431814 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:23.431797    5678 scope.go:122] "RemoveContainer" containerID="07a863ec3e008a3ed94dd22b5b8578c5a1b1b85e0a1ec9ebb7291f67d8fca16b"
Oct 30 22:21:32.179549 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:32.179482    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-65e59f/simulator-w4-65e59f-kserve-85698bcd47-przch" podUID="ee5e843b-17be-4623-8c95-dc07c8b7f506" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:21:32.333534 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:32.333485    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a00d9b/distinct-2-w1-a00d9b-kserve-68fd95957c-dmgnb" podUID="23a81fda-f7e3-4f91-8d96-9f19c7de6e09" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:21:32.348533 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:32.348494    5678 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a00d9b/distinct-2-w1-a00d9b-kserve-68fd95957c-dmgnb" podUID="23a81fda-f7e3-4f91-8d96-9f19c7de6e09" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:21:32.436481 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:32.435497    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9ndd9" podUID="81bd6415-f42f-446b-a64e-956f794b3956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:21:32.444720 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:32.443575    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-65e59f/premium-w4-65e59f-kserve-6dc49468f8-rlzhx" podUID="8590a783-eb24-4759-ad74-1369f36aa9d2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:21:32.504537 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:32.504490    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-65e59f/distinct-2-w4-65e59f-kserve-689566757d-bf98q" podUID="64e66fdb-056e-4086-a85a-8c02613584a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:21:32.505135 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:32.505104    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7bb1c2/simulator-w3-7bb1c2-kserve-675645869c-vcwqv" podUID="1fce9daf-68c9-41d0-825b-2012bb93c8ba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Oct 30 22:21:32.506821 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:32.506789    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7bb1c2/distinct-w3-7bb1c2-kserve-8d695596c-8zgtw" podUID="36f0a6a2-54be-49bf-9202-502960cbb134" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded"
Oct 30 22:21:32.588449 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:32.581929    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7bb1c2/premium-w3-7bb1c2-kserve-7dbf875d88-dqxn5" podUID="a58c911f-e2eb-4820-a6b6-24afe78b5fd4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:21:32.588449 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:32.583917    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb0b93/distinct-2-w6-fb0b93-kserve-7d65bb4698-m6tkh" podUID="6fd276ef-522b-4bb4-b101-0a9c3fe9ecd3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:21:32.588449 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:32.587977    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb0b93/simulator-w6-fb0b93-kserve-55b784b479-588bn" podUID="951a6f7e-a22e-40f2-ace2-1ce5d3bc9b49" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 30 22:21:32.616540 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:32.609473    5678 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-7bb1c2/distinct-w3-7bb1c2-kserve-8d695596c-8zgtw" podUID="36f0a6a2-54be-49bf-9202-502960cbb134" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:21:32.799826 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:32.799731    5678 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-7bb1c2/simulator-w3-7bb1c2-kserve-675645869c-vcwqv" podUID="1fce9daf-68c9-41d0-825b-2012bb93c8ba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": context deadline exceeded"
Oct 30 22:21:32.935559 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:32.935514    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779f32/premium-w0-779f32-kserve-67978797c8-jpkvl" podUID="40138fb9-9a75-4458-a351-11eb61c4a1e3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:21:33.237494 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:33.227584    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-65e59f/simulator-w4-65e59f-kserve-85698bcd47-przch" podUID="ee5e843b-17be-4623-8c95-dc07c8b7f506" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Oct 30 22:21:33.248213 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:33.245543    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-65e59f/simulator-w4-65e59f-kserve-85698bcd47-przch"
Oct 30 22:21:33.402466 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:33.354742    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a00d9b/distinct-2-w1-a00d9b-kserve-68fd95957c-dmgnb" podUID="23a81fda-f7e3-4f91-8d96-9f19c7de6e09" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:21:33.402466 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:33.355068    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a00d9b/distinct-2-w1-a00d9b-kserve-68fd95957c-dmgnb"
Oct 30 22:21:33.461428 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:33.450198    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-65e59f/premium-w4-65e59f-kserve-6dc49468f8-rlzhx" podUID="8590a783-eb24-4759-ad74-1369f36aa9d2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:21:33.469425 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:33.465636    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9ndd9" podUID="81bd6415-f42f-446b-a64e-956f794b3956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": context deadline exceeded"
Oct 30 22:21:33.500476 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:33.487285    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-65e59f/premium-w4-65e59f-kserve-6dc49468f8-rlzhx"
Oct 30 22:21:33.500476 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:33.487357    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9ndd9"
Oct 30 22:21:33.701427 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:33.634485    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7bb1c2/simulator-w3-7bb1c2-kserve-675645869c-vcwqv" podUID="1fce9daf-68c9-41d0-825b-2012bb93c8ba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Oct 30 22:21:33.701427 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:33.635437    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb0b93/distinct-2-w6-fb0b93-kserve-7d65bb4698-m6tkh" podUID="6fd276ef-522b-4bb4-b101-0a9c3fe9ecd3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:21:33.701427 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:33.635490    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb0b93/simulator-w6-fb0b93-kserve-55b784b479-588bn" podUID="951a6f7e-a22e-40f2-ace2-1ce5d3bc9b49" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 30 22:21:33.701427 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:33.649945    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-7bb1c2/simulator-w3-7bb1c2-kserve-675645869c-vcwqv"
Oct 30 22:21:33.701427 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:33.664470    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-fb0b93/distinct-2-w6-fb0b93-kserve-7d65bb4698-m6tkh"
Oct 30 22:21:33.701427 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:33.664585    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-fb0b93/simulator-w6-fb0b93-kserve-55b784b479-588bn"
Oct 30 22:21:33.814468 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:33.803484    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7bb1c2/premium-w3-7bb1c2-kserve-7dbf875d88-dqxn5" podUID="a58c911f-e2eb-4820-a6b6-24afe78b5fd4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:21:33.814468 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:33.804225    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-7bb1c2/premium-w3-7bb1c2-kserve-7dbf875d88-dqxn5"
Oct 30 22:21:33.868955 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:33.868926    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-fb0b93/simulator-w6-fb0b93-kserve-55b784b479-588bn"
Oct 30 22:21:33.869831 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:33.869811    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-fb0b93/distinct-2-w6-fb0b93-kserve-7d65bb4698-m6tkh"
Oct 30 22:21:33.915435 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:33.910034    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-7bb1c2/simulator-w3-7bb1c2-kserve-675645869c-vcwqv"
Oct 30 22:21:34.001576 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:34.001069    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a00d9b/distinct-2-w1-a00d9b-kserve-68fd95957c-dmgnb"
Oct 30 22:21:34.009012 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:34.006926    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779f32/premium-w0-779f32-kserve-67978797c8-jpkvl" podUID="40138fb9-9a75-4458-a351-11eb61c4a1e3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:21:34.009012 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:34.007613    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-779f32/premium-w0-779f32-kserve-67978797c8-jpkvl"
Oct 30 22:21:34.019717 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:34.019697    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9ndd9"
Oct 30 22:21:34.042492 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:34.042339    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-7bb1c2/premium-w3-7bb1c2-kserve-7dbf875d88-dqxn5"
Oct 30 22:21:34.063424 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:34.045233    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-779f32/premium-w0-779f32-kserve-67978797c8-jpkvl"
Oct 30 22:21:34.081033 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:34.080955    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-65e59f/simulator-w4-65e59f-kserve-85698bcd47-przch"
Oct 30 22:21:34.082558 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:34.082531    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-65e59f/premium-w4-65e59f-kserve-6dc49468f8-rlzhx"
Oct 30 22:21:36.765562 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:36.765520    5678 patch_prober.go:29] interesting pod/e2e-shared-a-w5-3cc1b3-openshift-default-744b6b9d7f-lx8n5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:21:36.766083 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:36.765582    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w5-3cc1b3-openshift-default-744b6b9d7f-lx8n5" podUID="4b6efa36-7840-418b-be4a-723ca4407a91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:21:39.963449 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:39.963389    5678 generic.go:356] "Generic (PLEG): container finished" podID="4b6efa36-7840-418b-be4a-723ca4407a91" containerID="0f413616c94f35de7224875324a6cc2aa3bc8255d48c60b79f78ca0fafddd51d" exitCode=0
Oct 30 22:21:39.963861 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:39.963449    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-3cc1b3-openshift-default-744b6b9d7f-lx8n5" event={"ID":"4b6efa36-7840-418b-be4a-723ca4407a91","Type":"ContainerDied","Data":"0f413616c94f35de7224875324a6cc2aa3bc8255d48c60b79f78ca0fafddd51d"}
Oct 30 22:21:39.963912 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:39.963900    5678 scope.go:122] "RemoveContainer" containerID="0f413616c94f35de7224875324a6cc2aa3bc8255d48c60b79f78ca0fafddd51d"
Oct 30 22:21:39.965890 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:39.965859    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:21:39.966015 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:39.965931    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:21:39.966015 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:39.965969    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:21:40.308366 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:40.308295    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d143f49-swg9z"]
Oct 30 22:21:40.308966 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:40.308952    5678 cpu_manager.go:414] "containerMap: removing container" podUID="defca364-7e29-483a-b948-d41e8bb31cd9" containerName="revoke-keys"
Oct 30 22:21:40.309021 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:40.308968    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="defca364-7e29-483a-b948-d41e8bb31cd9" containerName="revoke-keys"
Oct 30 22:21:40.309021 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:40.308987    5678 cpu_manager.go:414] "containerMap: removing container" podUID="669af24d-752d-407e-af9b-8ba5c02c40de" containerName="istio-proxy"
Oct 30 22:21:40.309021 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:40.308992    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="669af24d-752d-407e-af9b-8ba5c02c40de" containerName="istio-proxy"
Oct 30 22:21:40.309113 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:40.309088    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="defca364-7e29-483a-b948-d41e8bb31cd9" containerName="revoke-keys"
Oct 30 22:21:40.309113 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:40.309100    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="669af24d-752d-407e-af9b-8ba5c02c40de" containerName="istio-proxy"
Oct 30 22:21:40.312082 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:40.312062    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d143f49-swg9z"
Oct 30 22:21:40.365059 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:40.364811    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:21:40.365059 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:40.364956    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5rvcj\""
Oct 30 22:21:40.370528 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:40.370499    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d143f49-swg9z"]
Oct 30 22:21:40.394628 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:40.394595    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4f6xw\" (UniqueName: \"kubernetes.io/projected/ee597b98-9681-40c7-8af6-5484942d8094-kube-api-access-4f6xw\") pod \"maas-api-revoke-sub-7d143f49-swg9z\" (UID: \"ee597b98-9681-40c7-8af6-5484942d8094\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d143f49-swg9z"
Oct 30 22:21:40.394986 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:40.394958    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee597b98-9681-40c7-8af6-5484942d8094-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7d143f49-swg9z\" (UID: \"ee597b98-9681-40c7-8af6-5484942d8094\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d143f49-swg9z"
Oct 30 22:21:40.395115 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:40.395031    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee597b98-9681-40c7-8af6-5484942d8094-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7d143f49-swg9z\" (UID: \"ee597b98-9681-40c7-8af6-5484942d8094\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d143f49-swg9z"
Oct 30 22:21:40.496463 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:40.496430    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee597b98-9681-40c7-8af6-5484942d8094-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7d143f49-swg9z\" (UID: \"ee597b98-9681-40c7-8af6-5484942d8094\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d143f49-swg9z"
Oct 30 22:21:40.496632 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:40.496614    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4f6xw\" (UniqueName: \"kubernetes.io/projected/ee597b98-9681-40c7-8af6-5484942d8094-kube-api-access-4f6xw\") pod \"maas-api-revoke-sub-7d143f49-swg9z\" (UID: \"ee597b98-9681-40c7-8af6-5484942d8094\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d143f49-swg9z"
Oct 30 22:21:40.496759 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:40.496743    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee597b98-9681-40c7-8af6-5484942d8094-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7d143f49-swg9z\" (UID: \"ee597b98-9681-40c7-8af6-5484942d8094\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d143f49-swg9z"
Oct 30 22:21:40.497385 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:40.497358    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee597b98-9681-40c7-8af6-5484942d8094-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7d143f49-swg9z\" (UID: \"ee597b98-9681-40c7-8af6-5484942d8094\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d143f49-swg9z"
Oct 30 22:21:40.499005 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:40.498983    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee597b98-9681-40c7-8af6-5484942d8094-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7d143f49-swg9z\" (UID: \"ee597b98-9681-40c7-8af6-5484942d8094\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d143f49-swg9z"
Oct 30 22:21:40.561867 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:40.561775    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4f6xw\" (UniqueName: \"kubernetes.io/projected/ee597b98-9681-40c7-8af6-5484942d8094-kube-api-access-4f6xw\") pod \"maas-api-revoke-sub-7d143f49-swg9z\" (UID: \"ee597b98-9681-40c7-8af6-5484942d8094\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d143f49-swg9z"
Oct 30 22:21:40.584424 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:21:40.582106    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 22:21:40.640692 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:40.640648    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d143f49-swg9z"
Oct 30 22:21:40.973647 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:40.973604    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-3cc1b3-openshift-default-744b6b9d7f-lx8n5" event={"ID":"4b6efa36-7840-418b-be4a-723ca4407a91","Type":"ContainerStarted","Data":"d21d441a96db7f14edbbc29043f50740221659824026888da3bcaf666bc476de"}
Oct 30 22:21:41.055076 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:41.054689    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d143f49-swg9z"]
Oct 30 22:21:41.307499 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:41.307467    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-3cc1b3-openshift-default-744b6b9d7f-lx8n5"
Oct 30 22:21:41.307819 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:41.307801    5678 patch_prober.go:29] interesting pod/e2e-shared-a-w5-3cc1b3-openshift-default-744b6b9d7f-lx8n5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body=
Oct 30 22:21:41.307869 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:41.307848    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-3cc1b3-openshift-default-744b6b9d7f-lx8n5" podUID="4b6efa36-7840-418b-be4a-723ca4407a91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused"
Oct 30 22:21:41.982928 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:41.982888    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d143f49-swg9z" event={"ID":"ee597b98-9681-40c7-8af6-5484942d8094","Type":"ContainerStarted","Data":"16f17f2328b48909248b4edd4484d4ad73959ad20cbb3a8a73f946b4e86ce39d"}
Oct 30 22:21:41.983371 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:41.982939    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d143f49-swg9z" event={"ID":"ee597b98-9681-40c7-8af6-5484942d8094","Type":"ContainerStarted","Data":"78b01751328d36c02ba733f657a686d8e05d88583ddd3b014dcdc5d7f85833db"}
Oct 30 22:21:42.177422 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:42.164528    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d143f49-swg9z" podStartSLOduration=2.164511762 podStartE2EDuration="2.164511762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:21:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:21:42.164084667 +0000 UTC m=+1879.768392221" watchObservedRunningTime="2026-10-30 22:21:42.164511762 +0000 UTC m=+1879.768819316"
Oct 30 22:21:42.337422 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:42.307151    5678 patch_prober.go:29] interesting pod/e2e-shared-a-w5-3cc1b3-openshift-default-744b6b9d7f-lx8n5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body=
Oct 30 22:21:42.337422 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:42.307229    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-3cc1b3-openshift-default-744b6b9d7f-lx8n5" podUID="4b6efa36-7840-418b-be4a-723ca4407a91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused"
Oct 30 22:21:42.992523 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:42.992488    5678 generic.go:356] "Generic (PLEG): container finished" podID="ee597b98-9681-40c7-8af6-5484942d8094" containerID="16f17f2328b48909248b4edd4484d4ad73959ad20cbb3a8a73f946b4e86ce39d" exitCode=0
Oct 30 22:21:42.992913 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:42.992571    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d143f49-swg9z" event={"ID":"ee597b98-9681-40c7-8af6-5484942d8094","Type":"ContainerDied","Data":"16f17f2328b48909248b4edd4484d4ad73959ad20cbb3a8a73f946b4e86ce39d"}
Oct 30 22:21:43.307209 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:43.307123    5678 patch_prober.go:29] interesting pod/e2e-shared-a-w5-3cc1b3-openshift-default-744b6b9d7f-lx8n5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body=
Oct 30 22:21:43.307209 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:43.307175    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-3cc1b3-openshift-default-744b6b9d7f-lx8n5" podUID="4b6efa36-7840-418b-be4a-723ca4407a91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused"
Oct 30 22:21:44.249597 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:44.249569    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d143f49-swg9z"
Oct 30 22:21:44.306923 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:44.306831    5678 patch_prober.go:29] interesting pod/e2e-shared-a-w5-3cc1b3-openshift-default-744b6b9d7f-lx8n5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body=
Oct 30 22:21:44.306923 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:44.306887    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-3cc1b3-openshift-default-744b6b9d7f-lx8n5" podUID="4b6efa36-7840-418b-be4a-723ca4407a91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused"
Oct 30 22:21:44.312330 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:44.312303    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee597b98-9681-40c7-8af6-5484942d8094-kube-api-access-4f6xw\" (UniqueName: \"kubernetes.io/projected/ee597b98-9681-40c7-8af6-5484942d8094-kube-api-access-4f6xw\") pod \"ee597b98-9681-40c7-8af6-5484942d8094\" (UID: \"ee597b98-9681-40c7-8af6-5484942d8094\") "
Oct 30 22:21:44.312456 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:44.312438    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee597b98-9681-40c7-8af6-5484942d8094-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee597b98-9681-40c7-8af6-5484942d8094-maas-api-openshift-ca\") pod \"ee597b98-9681-40c7-8af6-5484942d8094\" (UID: \"ee597b98-9681-40c7-8af6-5484942d8094\") "
Oct 30 22:21:44.312753 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:44.312732    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee597b98-9681-40c7-8af6-5484942d8094-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee597b98-9681-40c7-8af6-5484942d8094-maas-api-xks-ca\") pod \"ee597b98-9681-40c7-8af6-5484942d8094\" (UID: \"ee597b98-9681-40c7-8af6-5484942d8094\") "
Oct 30 22:21:44.312823 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:44.312767    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee597b98-9681-40c7-8af6-5484942d8094-maas-api-openshift-ca" pod "ee597b98-9681-40c7-8af6-5484942d8094" (UID: "ee597b98-9681-40c7-8af6-5484942d8094"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:21:44.313263 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:44.313244    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee597b98-9681-40c7-8af6-5484942d8094-maas-api-openshift-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:21:44.314598 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:44.314575    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee597b98-9681-40c7-8af6-5484942d8094-kube-api-access-4f6xw" pod "ee597b98-9681-40c7-8af6-5484942d8094" (UID: "ee597b98-9681-40c7-8af6-5484942d8094"). InnerVolumeSpecName "kube-api-access-4f6xw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:21:44.314701 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:44.314618    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee597b98-9681-40c7-8af6-5484942d8094-maas-api-xks-ca" pod "ee597b98-9681-40c7-8af6-5484942d8094" (UID: "ee597b98-9681-40c7-8af6-5484942d8094"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:21:44.414564 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:44.414526    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee597b98-9681-40c7-8af6-5484942d8094-maas-api-xks-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:21:44.414564 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:44.414565    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4f6xw\" (UniqueName: \"kubernetes.io/projected/ee597b98-9681-40c7-8af6-5484942d8094-kube-api-access-4f6xw\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:21:45.009339 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:45.009309    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d143f49-swg9z"
Oct 30 22:21:45.009538 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:45.009305    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d143f49-swg9z" event={"ID":"ee597b98-9681-40c7-8af6-5484942d8094","Type":"ContainerDied","Data":"78b01751328d36c02ba733f657a686d8e05d88583ddd3b014dcdc5d7f85833db"}
Oct 30 22:21:45.009538 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:45.009443    5678 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="78b01751328d36c02ba733f657a686d8e05d88583ddd3b014dcdc5d7f85833db"
Oct 30 22:21:45.307714 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:45.307677    5678 patch_prober.go:29] interesting pod/e2e-shared-a-w5-3cc1b3-openshift-default-744b6b9d7f-lx8n5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body=
Oct 30 22:21:45.308141 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:45.307748    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-3cc1b3-openshift-default-744b6b9d7f-lx8n5" podUID="4b6efa36-7840-418b-be4a-723ca4407a91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused"
Oct 30 22:21:46.311334 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:46.311304    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-3cc1b3-openshift-default-744b6b9d7f-lx8n5"
Oct 30 22:21:46.311830 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:46.311626    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-3cc1b3-openshift-default-744b6b9d7f-lx8n5"
Oct 30 22:21:46.312317 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:46.312299    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-3cc1b3-openshift-default-744b6b9d7f-lx8n5"
Oct 30 22:21:59.660374 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:59.660340    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dafc795b-hrsc4"]
Oct 30 22:21:59.661152 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:59.661133    5678 cpu_manager.go:414] "containerMap: removing container" podUID="ee597b98-9681-40c7-8af6-5484942d8094" containerName="revoke-keys"
Oct 30 22:21:59.661152 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:59.661150    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee597b98-9681-40c7-8af6-5484942d8094" containerName="revoke-keys"
Oct 30 22:21:59.661365 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:59.661329    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee597b98-9681-40c7-8af6-5484942d8094" containerName="revoke-keys"
Oct 30 22:21:59.665223 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:59.665203    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dafc795b-hrsc4"
Oct 30 22:21:59.723155 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:59.723124    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dafc795b-hrsc4"]
Oct 30 22:21:59.743993 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:59.743963    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjdr5\" (UniqueName: \"kubernetes.io/projected/e4845576-736c-471b-83f1-ca535529773d-kube-api-access-vjdr5\") pod \"maas-api-revoke-sub-dafc795b-hrsc4\" (UID: \"e4845576-736c-471b-83f1-ca535529773d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dafc795b-hrsc4"
Oct 30 22:21:59.744168 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:59.744039    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4845576-736c-471b-83f1-ca535529773d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dafc795b-hrsc4\" (UID: \"e4845576-736c-471b-83f1-ca535529773d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dafc795b-hrsc4"
Oct 30 22:21:59.744236 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:59.744158    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4845576-736c-471b-83f1-ca535529773d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dafc795b-hrsc4\" (UID: \"e4845576-736c-471b-83f1-ca535529773d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dafc795b-hrsc4"
Oct 30 22:21:59.764867 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:59.764837    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:21:59.764991 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:59.764908    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5rvcj\""
Oct 30 22:21:59.846182 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:59.846150    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjdr5\" (UniqueName: \"kubernetes.io/projected/e4845576-736c-471b-83f1-ca535529773d-kube-api-access-vjdr5\") pod \"maas-api-revoke-sub-dafc795b-hrsc4\" (UID: \"e4845576-736c-471b-83f1-ca535529773d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dafc795b-hrsc4"
Oct 30 22:21:59.846350 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:59.846296    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4845576-736c-471b-83f1-ca535529773d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dafc795b-hrsc4\" (UID: \"e4845576-736c-471b-83f1-ca535529773d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dafc795b-hrsc4"
Oct 30 22:21:59.846350 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:59.846338    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4845576-736c-471b-83f1-ca535529773d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dafc795b-hrsc4\" (UID: \"e4845576-736c-471b-83f1-ca535529773d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dafc795b-hrsc4"
Oct 30 22:21:59.847004 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:59.846982    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4845576-736c-471b-83f1-ca535529773d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dafc795b-hrsc4\" (UID: \"e4845576-736c-471b-83f1-ca535529773d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dafc795b-hrsc4"
Oct 30 22:21:59.848630 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:59.848606    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4845576-736c-471b-83f1-ca535529773d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dafc795b-hrsc4\" (UID: \"e4845576-736c-471b-83f1-ca535529773d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dafc795b-hrsc4"
Oct 30 22:21:59.905056 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:59.905024    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjdr5\" (UniqueName: \"kubernetes.io/projected/e4845576-736c-471b-83f1-ca535529773d-kube-api-access-vjdr5\") pod \"maas-api-revoke-sub-dafc795b-hrsc4\" (UID: \"e4845576-736c-471b-83f1-ca535529773d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dafc795b-hrsc4"
Oct 30 22:21:59.979430 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:21:59.979337    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dafc795b-hrsc4"
Oct 30 22:22:00.369908 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:22:00.369873    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dafc795b-hrsc4"]
Oct 30 22:22:01.191389 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:22:01.191358    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dafc795b-hrsc4" event={"ID":"e4845576-736c-471b-83f1-ca535529773d","Type":"ContainerStarted","Data":"a13691238de04f3734df5801600f4c12527d5438c2227cab25e360d5c3adde48"}
Oct 30 22:22:01.191818 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:22:01.191449    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dafc795b-hrsc4" event={"ID":"e4845576-736c-471b-83f1-ca535529773d","Type":"ContainerStarted","Data":"2fc6b288cd443068a0f5f9d48b8437320577b880e40bff531f3a9dbc51b3a42f"}
Oct 30 22:22:01.307892 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:22:01.307841    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dafc795b-hrsc4" podStartSLOduration=2.3078261380000002 podStartE2EDuration="2.307826138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:21:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:22:01.305288863 +0000 UTC m=+1898.909596419" watchObservedRunningTime="2026-10-30 22:22:01.307826138 +0000 UTC m=+1898.912133691"
Oct 30 22:22:02.200284 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:22:02.200247    5678 generic.go:356] "Generic (PLEG): container finished" podID="e4845576-736c-471b-83f1-ca535529773d" containerID="a13691238de04f3734df5801600f4c12527d5438c2227cab25e360d5c3adde48" exitCode=0
Oct 30 22:22:02.200716 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:22:02.200331    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dafc795b-hrsc4" event={"ID":"e4845576-736c-471b-83f1-ca535529773d","Type":"ContainerDied","Data":"a13691238de04f3734df5801600f4c12527d5438c2227cab25e360d5c3adde48"}
Oct 30 22:22:03.423930 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:22:03.423889    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dafc795b-hrsc4"
Oct 30 22:22:03.520591 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:22:03.520508    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4845576-736c-471b-83f1-ca535529773d-kube-api-access-vjdr5\" (UniqueName: \"kubernetes.io/projected/e4845576-736c-471b-83f1-ca535529773d-kube-api-access-vjdr5\") pod \"e4845576-736c-471b-83f1-ca535529773d\" (UID: \"e4845576-736c-471b-83f1-ca535529773d\") "
Oct 30 22:22:03.520591 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:22:03.520582    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4845576-736c-471b-83f1-ca535529773d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4845576-736c-471b-83f1-ca535529773d-maas-api-openshift-ca\") pod \"e4845576-736c-471b-83f1-ca535529773d\" (UID: \"e4845576-736c-471b-83f1-ca535529773d\") "
Oct 30 22:22:03.520829 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:22:03.520625    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4845576-736c-471b-83f1-ca535529773d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4845576-736c-471b-83f1-ca535529773d-maas-api-xks-ca\") pod \"e4845576-736c-471b-83f1-ca535529773d\" (UID: \"e4845576-736c-471b-83f1-ca535529773d\") "
Oct 30 22:22:03.520959 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:22:03.520929    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4845576-736c-471b-83f1-ca535529773d-maas-api-openshift-ca" pod "e4845576-736c-471b-83f1-ca535529773d" (UID: "e4845576-736c-471b-83f1-ca535529773d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:22:03.521253 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:22:03.521228    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4845576-736c-471b-83f1-ca535529773d-maas-api-openshift-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:03.522950 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:22:03.522918    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4845576-736c-471b-83f1-ca535529773d-kube-api-access-vjdr5" pod "e4845576-736c-471b-83f1-ca535529773d" (UID: "e4845576-736c-471b-83f1-ca535529773d"). InnerVolumeSpecName "kube-api-access-vjdr5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:22:03.523326 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:22:03.523303    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4845576-736c-471b-83f1-ca535529773d-maas-api-xks-ca" pod "e4845576-736c-471b-83f1-ca535529773d" (UID: "e4845576-736c-471b-83f1-ca535529773d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:22:03.623497 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:22:03.623445    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjdr5\" (UniqueName: \"kubernetes.io/projected/e4845576-736c-471b-83f1-ca535529773d-kube-api-access-vjdr5\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:03.623497 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:22:03.623480    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4845576-736c-471b-83f1-ca535529773d-maas-api-xks-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:04.217976 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:22:04.217949    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dafc795b-hrsc4"
Oct 30 22:22:04.218147 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:22:04.217985    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dafc795b-hrsc4" event={"ID":"e4845576-736c-471b-83f1-ca535529773d","Type":"ContainerDied","Data":"2fc6b288cd443068a0f5f9d48b8437320577b880e40bff531f3a9dbc51b3a42f"}
Oct 30 22:22:04.218147 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:22:04.218038    5678 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2fc6b288cd443068a0f5f9d48b8437320577b880e40bff531f3a9dbc51b3a42f"
Oct 30 22:22:10.574427 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:22:10.574364    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 22:22:23.525705 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:22:23.525674    5678 scope.go:122] "RemoveContainer" containerID="89a9a36b09749a3312ec29b11c219826265df15a4d3a24bc9bda056f5cca3d38"
Oct 30 22:22:23.535657 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:22:23.535631    5678 scope.go:122] "RemoveContainer" containerID="948a3dbd938682d6ca7abf6a2a948bb2106cfeb1b08989d5ec85a3dac7617876"
Oct 30 22:22:23.544638 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:22:23.544615    5678 scope.go:122] "RemoveContainer" containerID="a3f9a432af4e2b0be712b48b2f0187fa8f12ddcff1719b9854d8cde91c50cf28"
Oct 30 22:22:36.737425 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:22:36.713501    5678 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9ndd9" podUID="81bd6415-f42f-446b-a64e-956f794b3956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:36.880375 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:22:36.878542    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb0b93/simulator-w6-fb0b93-kserve-55b784b479-588bn" podUID="951a6f7e-a22e-40f2-ace2-1ce5d3bc9b49" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:36.880375 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:22:36.878947    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb0b93/distinct-2-w6-fb0b93-kserve-7d65bb4698-m6tkh" podUID="6fd276ef-522b-4bb4-b101-0a9c3fe9ecd3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": context deadline exceeded"
Oct 30 22:22:36.919607 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:22:36.919555    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7bb1c2/simulator-w3-7bb1c2-kserve-675645869c-vcwqv" podUID="1fce9daf-68c9-41d0-825b-2012bb93c8ba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:37.092482 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:22:37.091498    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9ndd9" podUID="81bd6415-f42f-446b-a64e-956f794b3956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:37.092482 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:22:37.091883    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-65e59f/simulator-w4-65e59f-kserve-85698bcd47-przch" podUID="ee5e843b-17be-4623-8c95-dc07c8b7f506" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Oct 30 22:22:37.092482 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:22:37.092182    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779f32/premium-w0-779f32-kserve-67978797c8-jpkvl" podUID="40138fb9-9a75-4458-a351-11eb61c4a1e3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Oct 30 22:22:37.102791 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:22:37.102752    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-65e59f/premium-w4-65e59f-kserve-6dc49468f8-rlzhx" podUID="8590a783-eb24-4759-ad74-1369f36aa9d2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:37.118521 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:22:37.118478    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a00d9b/distinct-2-w1-a00d9b-kserve-68fd95957c-dmgnb" podUID="23a81fda-f7e3-4f91-8d96-9f19c7de6e09" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:40.573258 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:22:40.573216    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 22:22:50.998917 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:22:50.998881    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-vzfdw"]
Oct 30 22:22:50.999338 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:22:50.999093    5678 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-vzfdw" podUID="24750677-9d8f-46f4-9f0e-acc3d230fb8f" containerName="istio-proxy" containerID="cri-o://e9bc67d5b4029824f09867d4654c397405e4d862427cc702d0ac4bb628896208" gracePeriod=30
Oct 30 22:22:55.056334 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:22:55.056301    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f198c49a-b2nsb"]
Oct 30 22:22:55.100992 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:22:55.100964    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f198c49a-b2nsb"]
Oct 30 22:22:56.142558 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:22:56.142526    5678 patch_prober.go:29] interesting pod/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-vzfdw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body=
Oct 30 22:22:56.142904 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:22:56.142589    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-vzfdw" podUID="24750677-9d8f-46f4-9f0e-acc3d230fb8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused"
Oct 30 22:22:56.271861 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:22:56.271801    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-vzfdw"
Oct 30 22:22:56.347300 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:22:56.347269    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24750677-9d8f-46f4-9f0e-acc3d230fb8f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24750677-9d8f-46f4-9f0e-acc3d230fb8f-credential-socket\") pod \"24750677-9d8f-46f4-9f0e-acc3d230fb8f\" (UID: \"24750677-9d8f-46f4-9f0e-acc3d230fb8f\") "
Oct 30 22:22:56.347530 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:22:56.347347    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24750677-9d8f-46f4-9f0e-acc3d230fb8f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24750677-9d8f-46f4-9f0e-acc3d230fb8f-workload-certs\") pod \"24750677-9d8f-46f4-9f0e-acc3d230fb8f\" (UID: \"24750677-9d8f-46f4-9f0e-acc3d230fb8f\") "
Oct 30 22:22:56.347530 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:22:56.347449    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24750677-9d8f-46f4-9f0e-acc3d230fb8f-kube-api-access-5mvvm\" (UniqueName: \"kubernetes.io/projected/24750677-9d8f-46f4-9f0e-acc3d230fb8f-kube-api-access-5mvvm\") pod \"24750677-9d8f-46f4-9f0e-acc3d230fb8f\" (UID: \"24750677-9d8f-46f4-9f0e-acc3d230fb8f\") "
Oct 30 22:22:56.347654 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:22:56.347594    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24750677-9d8f-46f4-9f0e-acc3d230fb8f-credential-socket" pod "24750677-9d8f-46f4-9f0e-acc3d230fb8f" (UID: "24750677-9d8f-46f4-9f0e-acc3d230fb8f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:22:56.347654 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:22:56.347610    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24750677-9d8f-46f4-9f0e-acc3d230fb8f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24750677-9d8f-46f4-9f0e-acc3d230fb8f-istio-envoy\") pod \"24750677-9d8f-46f4-9f0e-acc3d230fb8f\" (UID: \"24750677-9d8f-46f4-9f0e-acc3d230fb8f\") "
Oct 30 22:22:56.347770 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:22:56.347655    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24750677-9d8f-46f4-9f0e-acc3d230fb8f-workload-certs" pod "24750677-9d8f-46f4-9f0e-acc3d230fb8f" (UID: "24750677-9d8f-46f4-9f0e-acc3d230fb8f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:22:56.347770 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:22:56.347680    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24750677-9d8f-46f4-9f0e-acc3d230fb8f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24750677-9d8f-46f4-9f0e-acc3d230fb8f-workload-socket\") pod \"24750677-9d8f-46f4-9f0e-acc3d230fb8f\" (UID: \"24750677-9d8f-46f4-9f0e-acc3d230fb8f\") "
Oct 30 22:22:56.347770 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:22:56.347724    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24750677-9d8f-46f4-9f0e-acc3d230fb8f-istio-token\" (UniqueName: \"kubernetes.io/projected/24750677-9d8f-46f4-9f0e-acc3d230fb8f-istio-token\") pod \"24750677-9d8f-46f4-9f0e-acc3d230fb8f\" (UID: \"24750677-9d8f-46f4-9f0e-acc3d230fb8f\") "
Oct 30 22:22:56.347770 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:22:56.347753    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24750677-9d8f-46f4-9f0e-acc3d230fb8f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24750677-9d8f-46f4-9f0e-acc3d230fb8f-istiod-ca-cert\") pod \"24750677-9d8f-46f4-9f0e-acc3d230fb8f\" (UID: \"24750677-9d8f-46f4-9f0e-acc3d230fb8f\") "
Oct 30 22:22:56.347969 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:22:56.347870    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/24750677-9d8f-46f4-9f0e-acc3d230fb8f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24750677-9d8f-46f4-9f0e-acc3d230fb8f-istio-podinfo\") pod \"24750677-9d8f-46f4-9f0e-acc3d230fb8f\" (UID: \"24750677-9d8f-46f4-9f0e-acc3d230fb8f\") "
Oct 30 22:22:56.347969 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:22:56.347936    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24750677-9d8f-46f4-9f0e-acc3d230fb8f-workload-socket" pod "24750677-9d8f-46f4-9f0e-acc3d230fb8f" (UID: "24750677-9d8f-46f4-9f0e-acc3d230fb8f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:22:56.347969 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:22:56.347944    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24750677-9d8f-46f4-9f0e-acc3d230fb8f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24750677-9d8f-46f4-9f0e-acc3d230fb8f-istio-data\") pod \"24750677-9d8f-46f4-9f0e-acc3d230fb8f\" (UID: \"24750677-9d8f-46f4-9f0e-acc3d230fb8f\") "
Oct 30 22:22:56.348229 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:22:56.348087    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24750677-9d8f-46f4-9f0e-acc3d230fb8f-istiod-ca-cert" pod "24750677-9d8f-46f4-9f0e-acc3d230fb8f" (UID: "24750677-9d8f-46f4-9f0e-acc3d230fb8f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:22:56.348229 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:22:56.348155    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24750677-9d8f-46f4-9f0e-acc3d230fb8f-istio-data" pod "24750677-9d8f-46f4-9f0e-acc3d230fb8f" (UID: "24750677-9d8f-46f4-9f0e-acc3d230fb8f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:22:56.348700 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:22:56.348676    5678 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24750677-9d8f-46f4-9f0e-acc3d230fb8f-workload-socket\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:56.348772 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:22:56.348707    5678 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24750677-9d8f-46f4-9f0e-acc3d230fb8f-istiod-ca-cert\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:56.348772 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:22:56.348724    5678 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24750677-9d8f-46f4-9f0e-acc3d230fb8f-istio-data\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:56.348772 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:22:56.348739    5678 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24750677-9d8f-46f4-9f0e-acc3d230fb8f-credential-socket\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:56.348772 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:22:56.348754    5678 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24750677-9d8f-46f4-9f0e-acc3d230fb8f-workload-certs\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:56.349822 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:22:56.349804    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24750677-9d8f-46f4-9f0e-acc3d230fb8f-istio-token" pod "24750677-9d8f-46f4-9f0e-acc3d230fb8f" (UID: "24750677-9d8f-46f4-9f0e-acc3d230fb8f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:22:56.349953 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:22:56.349938    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/24750677-9d8f-46f4-9f0e-acc3d230fb8f-istio-podinfo" pod "24750677-9d8f-46f4-9f0e-acc3d230fb8f" (UID: "24750677-9d8f-46f4-9f0e-acc3d230fb8f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:22:56.350130 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:22:56.350110    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24750677-9d8f-46f4-9f0e-acc3d230fb8f-istio-envoy" pod "24750677-9d8f-46f4-9f0e-acc3d230fb8f" (UID: "24750677-9d8f-46f4-9f0e-acc3d230fb8f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:22:56.350206 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:22:56.350183    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24750677-9d8f-46f4-9f0e-acc3d230fb8f-kube-api-access-5mvvm" pod "24750677-9d8f-46f4-9f0e-acc3d230fb8f" (UID: "24750677-9d8f-46f4-9f0e-acc3d230fb8f"). InnerVolumeSpecName "kube-api-access-5mvvm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:22:56.450192 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:22:56.450163    5678 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24750677-9d8f-46f4-9f0e-acc3d230fb8f-istio-podinfo\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:56.450192 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:22:56.450190    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mvvm\" (UniqueName: \"kubernetes.io/projected/24750677-9d8f-46f4-9f0e-acc3d230fb8f-kube-api-access-5mvvm\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:56.450385 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:22:56.450201    5678 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24750677-9d8f-46f4-9f0e-acc3d230fb8f-istio-envoy\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:56.450385 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:22:56.450213    5678 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24750677-9d8f-46f4-9f0e-acc3d230fb8f-istio-token\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:56.525593 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:22:56.525519    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa29d2ac-60f7-4a90-8d2c-4161eccb6a1d" path="/var/lib/kubelet/pods/aa29d2ac-60f7-4a90-8d2c-4161eccb6a1d/volumes"
Oct 30 22:22:56.937126 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:22:56.937078    5678 generic.go:356] "Generic (PLEG): container finished" podID="24750677-9d8f-46f4-9f0e-acc3d230fb8f" containerID="e9bc67d5b4029824f09867d4654c397405e4d862427cc702d0ac4bb628896208" exitCode=0
Oct 30 22:22:56.937126 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:22:56.937133    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-vzfdw"
Oct 30 22:22:56.937362 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:22:56.937166    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-vzfdw" event={"ID":"24750677-9d8f-46f4-9f0e-acc3d230fb8f","Type":"ContainerDied","Data":"e9bc67d5b4029824f09867d4654c397405e4d862427cc702d0ac4bb628896208"}
Oct 30 22:22:56.937362 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:22:56.937224    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-vzfdw" event={"ID":"24750677-9d8f-46f4-9f0e-acc3d230fb8f","Type":"ContainerDied","Data":"62851d4513aef519293e952521ee38c3e61228efc0787c3c8bbcb31de52972a7"}
Oct 30 22:22:56.937362 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:22:56.937253    5678 scope.go:122] "RemoveContainer" containerID="e9bc67d5b4029824f09867d4654c397405e4d862427cc702d0ac4bb628896208"
Oct 30 22:22:56.948224 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:22:56.948205    5678 scope.go:122] "RemoveContainer" containerID="e9bc67d5b4029824f09867d4654c397405e4d862427cc702d0ac4bb628896208"
Oct 30 22:22:56.948536 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:22:56.948518    5678 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9bc67d5b4029824f09867d4654c397405e4d862427cc702d0ac4bb628896208\": container with ID starting with e9bc67d5b4029824f09867d4654c397405e4d862427cc702d0ac4bb628896208 not found: ID does not exist" containerID="e9bc67d5b4029824f09867d4654c397405e4d862427cc702d0ac4bb628896208"
Oct 30 22:22:56.948608 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:22:56.948543    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9bc67d5b4029824f09867d4654c397405e4d862427cc702d0ac4bb628896208"} err="failed to get container status \"e9bc67d5b4029824f09867d4654c397405e4d862427cc702d0ac4bb628896208\": rpc error: code = NotFound desc = could not find container \"e9bc67d5b4029824f09867d4654c397405e4d862427cc702d0ac4bb628896208\": container with ID starting with e9bc67d5b4029824f09867d4654c397405e4d862427cc702d0ac4bb628896208 not found: ID does not exist"
Oct 30 22:22:57.149514 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:22:57.149481    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-vzfdw"]
Oct 30 22:22:57.178639 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:22:57.178603    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-vzfdw"]
Oct 30 22:22:58.525621 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:22:58.525590    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24750677-9d8f-46f4-9f0e-acc3d230fb8f" path="/var/lib/kubelet/pods/24750677-9d8f-46f4-9f0e-acc3d230fb8f/volumes"
Oct 30 22:23:07.503066 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:07.503035    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-7f76c4-79fc59665d-fhwbf"]
Oct 30 22:23:07.503763 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:07.503745    5678 cpu_manager.go:414] "containerMap: removing container" podUID="e4845576-736c-471b-83f1-ca535529773d" containerName="revoke-keys"
Oct 30 22:23:07.503811 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:07.503765    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4845576-736c-471b-83f1-ca535529773d" containerName="revoke-keys"
Oct 30 22:23:07.503811 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:07.503778    5678 cpu_manager.go:414] "containerMap: removing container" podUID="24750677-9d8f-46f4-9f0e-acc3d230fb8f" containerName="istio-proxy"
Oct 30 22:23:07.503811 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:07.503783    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24750677-9d8f-46f4-9f0e-acc3d230fb8f" containerName="istio-proxy"
Oct 30 22:23:07.503906 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:07.503886    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4845576-736c-471b-83f1-ca535529773d" containerName="revoke-keys"
Oct 30 22:23:07.503906 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:07.503895    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="24750677-9d8f-46f4-9f0e-acc3d230fb8f" containerName="istio-proxy"
Oct 30 22:23:07.508862 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:07.508842    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7f76c4-79fc59665d-fhwbf"
Oct 30 22:23:07.559619 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:07.559580    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-7f76c4-serving-cert\""
Oct 30 22:23:07.564455 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:07.564429    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-7f76c4-79fc59665d-fhwbf"]
Oct 30 22:23:07.605361 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:07.605325    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/900b0bd8-1f49-4ce5-ad56-16b891688876-metrics-certs\") pod \"maas-api-e2e-api-a-7f76c4-79fc59665d-fhwbf\" (UID: \"900b0bd8-1f49-4ce5-ad56-16b891688876\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7f76c4-79fc59665d-fhwbf"
Oct 30 22:23:07.605505 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:07.605432    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/900b0bd8-1f49-4ce5-ad56-16b891688876-maas-api-tls\") pod \"maas-api-e2e-api-a-7f76c4-79fc59665d-fhwbf\" (UID: \"900b0bd8-1f49-4ce5-ad56-16b891688876\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7f76c4-79fc59665d-fhwbf"
Oct 30 22:23:07.605505 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:07.605480    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/900b0bd8-1f49-4ce5-ad56-16b891688876-openshift-service-ca\") pod \"maas-api-e2e-api-a-7f76c4-79fc59665d-fhwbf\" (UID: \"900b0bd8-1f49-4ce5-ad56-16b891688876\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7f76c4-79fc59665d-fhwbf"
Oct 30 22:23:07.605943 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:07.605908    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnmgf\" (UniqueName: \"kubernetes.io/projected/900b0bd8-1f49-4ce5-ad56-16b891688876-kube-api-access-dnmgf\") pod \"maas-api-e2e-api-a-7f76c4-79fc59665d-fhwbf\" (UID: \"900b0bd8-1f49-4ce5-ad56-16b891688876\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7f76c4-79fc59665d-fhwbf"
Oct 30 22:23:07.707374 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:07.707345    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/900b0bd8-1f49-4ce5-ad56-16b891688876-metrics-certs\") pod \"maas-api-e2e-api-a-7f76c4-79fc59665d-fhwbf\" (UID: \"900b0bd8-1f49-4ce5-ad56-16b891688876\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7f76c4-79fc59665d-fhwbf"
Oct 30 22:23:07.707534 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:07.707429    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/900b0bd8-1f49-4ce5-ad56-16b891688876-maas-api-tls\") pod \"maas-api-e2e-api-a-7f76c4-79fc59665d-fhwbf\" (UID: \"900b0bd8-1f49-4ce5-ad56-16b891688876\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7f76c4-79fc59665d-fhwbf"
Oct 30 22:23:07.707534 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:07.707480    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/900b0bd8-1f49-4ce5-ad56-16b891688876-openshift-service-ca\") pod \"maas-api-e2e-api-a-7f76c4-79fc59665d-fhwbf\" (UID: \"900b0bd8-1f49-4ce5-ad56-16b891688876\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7f76c4-79fc59665d-fhwbf"
Oct 30 22:23:07.707613 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:23:07.707586    5678 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-7f76c4-serving-cert: secret "maas-api-e2e-api-a-7f76c4-serving-cert" not found
Oct 30 22:23:07.707673 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:23:07.707663    5678 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/900b0bd8-1f49-4ce5-ad56-16b891688876-maas-api-tls podName:900b0bd8-1f49-4ce5-ad56-16b891688876 nodeName:}" failed. No retries permitted until 2026-10-30 22:23:08.207643899 +0000 UTC m=+1965.811951448 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/900b0bd8-1f49-4ce5-ad56-16b891688876-maas-api-tls") pod "maas-api-e2e-api-a-7f76c4-79fc59665d-fhwbf" (UID: "900b0bd8-1f49-4ce5-ad56-16b891688876") : secret "maas-api-e2e-api-a-7f76c4-serving-cert" not found
Oct 30 22:23:07.707728 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:07.707712    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnmgf\" (UniqueName: \"kubernetes.io/projected/900b0bd8-1f49-4ce5-ad56-16b891688876-kube-api-access-dnmgf\") pod \"maas-api-e2e-api-a-7f76c4-79fc59665d-fhwbf\" (UID: \"900b0bd8-1f49-4ce5-ad56-16b891688876\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7f76c4-79fc59665d-fhwbf"
Oct 30 22:23:07.708117 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:07.708088    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/900b0bd8-1f49-4ce5-ad56-16b891688876-openshift-service-ca\") pod \"maas-api-e2e-api-a-7f76c4-79fc59665d-fhwbf\" (UID: \"900b0bd8-1f49-4ce5-ad56-16b891688876\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7f76c4-79fc59665d-fhwbf"
Oct 30 22:23:07.709839 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:07.709817    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/900b0bd8-1f49-4ce5-ad56-16b891688876-metrics-certs\") pod \"maas-api-e2e-api-a-7f76c4-79fc59665d-fhwbf\" (UID: \"900b0bd8-1f49-4ce5-ad56-16b891688876\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7f76c4-79fc59665d-fhwbf"
Oct 30 22:23:07.772478 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:07.771537    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnmgf\" (UniqueName: \"kubernetes.io/projected/900b0bd8-1f49-4ce5-ad56-16b891688876-kube-api-access-dnmgf\") pod \"maas-api-e2e-api-a-7f76c4-79fc59665d-fhwbf\" (UID: \"900b0bd8-1f49-4ce5-ad56-16b891688876\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7f76c4-79fc59665d-fhwbf"
Oct 30 22:23:08.217396 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:08.217349    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/900b0bd8-1f49-4ce5-ad56-16b891688876-maas-api-tls\") pod \"maas-api-e2e-api-a-7f76c4-79fc59665d-fhwbf\" (UID: \"900b0bd8-1f49-4ce5-ad56-16b891688876\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7f76c4-79fc59665d-fhwbf"
Oct 30 22:23:08.219787 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:08.219767    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/900b0bd8-1f49-4ce5-ad56-16b891688876-maas-api-tls\") pod \"maas-api-e2e-api-a-7f76c4-79fc59665d-fhwbf\" (UID: \"900b0bd8-1f49-4ce5-ad56-16b891688876\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7f76c4-79fc59665d-fhwbf"
Oct 30 22:23:08.422602 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:08.422568    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7f76c4-79fc59665d-fhwbf"
Oct 30 22:23:08.807120 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:23:08.807088    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod900b0bd8_1f49_4ce5_ad56_16b891688876.slice/crio-6653e4295385e2dbbe7bfa4df6fc25a61eb83b12aed32a0a31ae306a43d12abe WatchSource:0}: Error finding container 6653e4295385e2dbbe7bfa4df6fc25a61eb83b12aed32a0a31ae306a43d12abe: Status 404 returned error can't find the container with id 6653e4295385e2dbbe7bfa4df6fc25a61eb83b12aed32a0a31ae306a43d12abe
Oct 30 22:23:08.807530 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:08.807503    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-7f76c4-79fc59665d-fhwbf"]
Oct 30 22:23:08.809102 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:08.809080    5678 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 22:23:09.031079 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:09.031038    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7f76c4-79fc59665d-fhwbf" event={"ID":"900b0bd8-1f49-4ce5-ad56-16b891688876","Type":"ContainerStarted","Data":"6653e4295385e2dbbe7bfa4df6fc25a61eb83b12aed32a0a31ae306a43d12abe"}
Oct 30 22:23:10.040077 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:10.039981    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7f76c4-79fc59665d-fhwbf" event={"ID":"900b0bd8-1f49-4ce5-ad56-16b891688876","Type":"ContainerStarted","Data":"c4c08c39286ffe02d5b417dfcefde789b26d1c08a4908d0f9481398565b8f268"}
Oct 30 22:23:10.040459 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:10.040078    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7f76c4-79fc59665d-fhwbf"
Oct 30 22:23:10.183897 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:10.183839    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7f76c4-79fc59665d-fhwbf" podStartSLOduration=2.279171623 podStartE2EDuration="3.183824631s" totalImagesPullingTime="904.653008ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:23:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:23:10.182642027 +0000 UTC m=+1967.786949580" watchObservedRunningTime="2026-10-30 22:23:10.183824631 +0000 UTC m=+1967.788132216"
Oct 30 22:23:10.582976 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:23:10.582942    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 22:23:16.053177 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:16.053143    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7f76c4-79fc59665d-fhwbf"
Oct 30 22:23:22.980641 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:22.980602    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-q828p"]
Oct 30 22:23:22.981060 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:22.980859    5678 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-q828p" podUID="4e8c9108-98ce-4a09-a8a9-f13def9228df" containerName="istio-proxy" containerID="cri-o://6850f4ac14eb5dfdc02469456f3c2aaae042e5fe7ab85bbc8fa45a39e4bd3435" gracePeriod=30
Oct 30 22:23:23.035866 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:23.035833    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-fpsll"]
Oct 30 22:23:23.036324 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:23.036270    5678 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-fpsll" podUID="52992d7b-821d-4b13-945c-d0b9b59ddc58" containerName="istio-proxy" containerID="cri-o://baa9ce10b5014f02d6861aa75dfc8069cbb7fa6cf3d720e9bdd79818ffa1c287" gracePeriod=30
Oct 30 22:23:23.086330 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:23.086301    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-88d9s"]
Oct 30 22:23:23.086629 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:23.086557    5678 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-88d9s" podUID="76d2c82e-13ed-497c-acbc-7748bfb133c7" containerName="discovery" containerID="cri-o://19944a2586d1a0f46f77065daa511ea819366a8510d92908d3a918724cf6a8f5" gracePeriod=30
Oct 30 22:23:23.123049 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:23.123021    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ldgt7"]
Oct 30 22:23:23.123452 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:23.123250    5678 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ldgt7" podUID="0e17f2f4-e217-43cf-81ba-2bfb2095dbd6" containerName="istio-proxy" containerID="cri-o://21e5075edd24d61f4ec91b159a9a0ffa69bdae573523d0dc2eeff217c8a37caf" gracePeriod=30
Oct 30 22:23:23.298661 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:23.298628    5678 generic.go:356] "Generic (PLEG): container finished" podID="76d2c82e-13ed-497c-acbc-7748bfb133c7" containerID="19944a2586d1a0f46f77065daa511ea819366a8510d92908d3a918724cf6a8f5" exitCode=0
Oct 30 22:23:23.298826 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:23.298807    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-88d9s" event={"ID":"76d2c82e-13ed-497c-acbc-7748bfb133c7","Type":"ContainerDied","Data":"19944a2586d1a0f46f77065daa511ea819366a8510d92908d3a918724cf6a8f5"}
Oct 30 22:23:23.379371 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:23.379346    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-88d9s"
Oct 30 22:23:23.463372 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:23.463343    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76d2c82e-13ed-497c-acbc-7748bfb133c7-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/76d2c82e-13ed-497c-acbc-7748bfb133c7-istio-csr-ca-configmap\") pod \"76d2c82e-13ed-497c-acbc-7748bfb133c7\" (UID: \"76d2c82e-13ed-497c-acbc-7748bfb133c7\") "
Oct 30 22:23:23.463564 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:23.463453    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/76d2c82e-13ed-497c-acbc-7748bfb133c7-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/76d2c82e-13ed-497c-acbc-7748bfb133c7-istio-csr-dns-cert\") pod \"76d2c82e-13ed-497c-acbc-7748bfb133c7\" (UID: \"76d2c82e-13ed-497c-acbc-7748bfb133c7\") "
Oct 30 22:23:23.463650 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:23.463637    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76d2c82e-13ed-497c-acbc-7748bfb133c7-kube-api-access-2jdz9\" (UniqueName: \"kubernetes.io/projected/76d2c82e-13ed-497c-acbc-7748bfb133c7-kube-api-access-2jdz9\") pod \"76d2c82e-13ed-497c-acbc-7748bfb133c7\" (UID: \"76d2c82e-13ed-497c-acbc-7748bfb133c7\") "
Oct 30 22:23:23.463739 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:23.463710    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76d2c82e-13ed-497c-acbc-7748bfb133c7-istio-token\" (UniqueName: \"kubernetes.io/projected/76d2c82e-13ed-497c-acbc-7748bfb133c7-istio-token\") pod \"76d2c82e-13ed-497c-acbc-7748bfb133c7\" (UID: \"76d2c82e-13ed-497c-acbc-7748bfb133c7\") "
Oct 30 22:23:23.463739 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:23.463726    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76d2c82e-13ed-497c-acbc-7748bfb133c7-istio-csr-ca-configmap" pod "76d2c82e-13ed-497c-acbc-7748bfb133c7" (UID: "76d2c82e-13ed-497c-acbc-7748bfb133c7"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:23:23.463837 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:23.463819    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76d2c82e-13ed-497c-acbc-7748bfb133c7-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/76d2c82e-13ed-497c-acbc-7748bfb133c7-local-certs\") pod \"76d2c82e-13ed-497c-acbc-7748bfb133c7\" (UID: \"76d2c82e-13ed-497c-acbc-7748bfb133c7\") "
Oct 30 22:23:23.463888 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:23.463878    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/76d2c82e-13ed-497c-acbc-7748bfb133c7-cacerts\" (UniqueName: \"kubernetes.io/secret/76d2c82e-13ed-497c-acbc-7748bfb133c7-cacerts\") pod \"76d2c82e-13ed-497c-acbc-7748bfb133c7\" (UID: \"76d2c82e-13ed-497c-acbc-7748bfb133c7\") "
Oct 30 22:23:23.464041 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:23.464006    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/76d2c82e-13ed-497c-acbc-7748bfb133c7-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/76d2c82e-13ed-497c-acbc-7748bfb133c7-istio-kubeconfig\") pod \"76d2c82e-13ed-497c-acbc-7748bfb133c7\" (UID: \"76d2c82e-13ed-497c-acbc-7748bfb133c7\") "
Oct 30 22:23:23.464917 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:23.464890    5678 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/76d2c82e-13ed-497c-acbc-7748bfb133c7-istio-csr-ca-configmap\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:23.465842 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:23.465812    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76d2c82e-13ed-497c-acbc-7748bfb133c7-istio-csr-dns-cert" pod "76d2c82e-13ed-497c-acbc-7748bfb133c7" (UID: "76d2c82e-13ed-497c-acbc-7748bfb133c7"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:23:23.466026 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:23.466003    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76d2c82e-13ed-497c-acbc-7748bfb133c7-local-certs" pod "76d2c82e-13ed-497c-acbc-7748bfb133c7" (UID: "76d2c82e-13ed-497c-acbc-7748bfb133c7"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:23:23.466213 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:23.466193    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76d2c82e-13ed-497c-acbc-7748bfb133c7-istio-token" pod "76d2c82e-13ed-497c-acbc-7748bfb133c7" (UID: "76d2c82e-13ed-497c-acbc-7748bfb133c7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:23:23.466297 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:23.466277    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76d2c82e-13ed-497c-acbc-7748bfb133c7-kube-api-access-2jdz9" pod "76d2c82e-13ed-497c-acbc-7748bfb133c7" (UID: "76d2c82e-13ed-497c-acbc-7748bfb133c7"). InnerVolumeSpecName "kube-api-access-2jdz9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:23:23.466362 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:23.466354    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76d2c82e-13ed-497c-acbc-7748bfb133c7-cacerts" pod "76d2c82e-13ed-497c-acbc-7748bfb133c7" (UID: "76d2c82e-13ed-497c-acbc-7748bfb133c7"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:23:23.466494 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:23.466468    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76d2c82e-13ed-497c-acbc-7748bfb133c7-istio-kubeconfig" pod "76d2c82e-13ed-497c-acbc-7748bfb133c7" (UID: "76d2c82e-13ed-497c-acbc-7748bfb133c7"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:23:23.566525 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:23.566464    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jdz9\" (UniqueName: \"kubernetes.io/projected/76d2c82e-13ed-497c-acbc-7748bfb133c7-kube-api-access-2jdz9\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:23.566525 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:23.566487    5678 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76d2c82e-13ed-497c-acbc-7748bfb133c7-istio-token\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:23.566525 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:23.566496    5678 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/76d2c82e-13ed-497c-acbc-7748bfb133c7-local-certs\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:23.566525 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:23.566504    5678 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/76d2c82e-13ed-497c-acbc-7748bfb133c7-cacerts\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:23.566525 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:23.566514    5678 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/76d2c82e-13ed-497c-acbc-7748bfb133c7-istio-kubeconfig\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:23.566525 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:23.566523    5678 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/76d2c82e-13ed-497c-acbc-7748bfb133c7-istio-csr-dns-cert\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:24.306807 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:24.306773    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-88d9s"
Oct 30 22:23:24.307229 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:24.306804    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-88d9s" event={"ID":"76d2c82e-13ed-497c-acbc-7748bfb133c7","Type":"ContainerDied","Data":"7bbf67b43e173ffe70189ec8cf04b8d45d2e248025e4b04f9e69bdb4534dc788"}
Oct 30 22:23:24.307229 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:24.306865    5678 scope.go:122] "RemoveContainer" containerID="19944a2586d1a0f46f77065daa511ea819366a8510d92908d3a918724cf6a8f5"
Oct 30 22:23:24.540106 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:24.540071    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-88d9s"]
Oct 30 22:23:24.590629 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:24.590543    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-88d9s"]
Oct 30 22:23:24.752414 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:24.752371    5678 patch_prober.go:29] interesting pod/e2e-worker-w2-bc714d-openshift-default-67558b95cf-fpsll container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:23:24.752570 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:24.752447    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-fpsll" podUID="52992d7b-821d-4b13-945c-d0b9b59ddc58" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:23:26.525463 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:26.525433    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76d2c82e-13ed-497c-acbc-7748bfb133c7" path="/var/lib/kubelet/pods/76d2c82e-13ed-497c-acbc-7748bfb133c7/volumes"
Oct 30 22:23:26.661429 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:26.660991    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-c24d61-openshift-default-7c56bd489-c85l8"]
Oct 30 22:23:26.664169 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:26.664139    5678 cpu_manager.go:414] "containerMap: removing container" podUID="76d2c82e-13ed-497c-acbc-7748bfb133c7" containerName="discovery"
Oct 30 22:23:26.665820 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:26.665797    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76d2c82e-13ed-497c-acbc-7748bfb133c7" containerName="discovery"
Oct 30 22:23:26.666234 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:26.666199    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="76d2c82e-13ed-497c-acbc-7748bfb133c7" containerName="discovery"
Oct 30 22:23:26.674964 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:26.674048    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c24d61-openshift-default-7c56bd489-c85l8"
Oct 30 22:23:26.735422 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:26.734773    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-c24d61-openshift-default-dockercfg-5ccnp\""
Oct 30 22:23:26.751641 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:26.751590    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd8986aa-b616-4a9e-9dd9-ae7eefd3e847-workload-socket\") pod \"e2e-api-a-c24d61-openshift-default-7c56bd489-c85l8\" (UID: \"bd8986aa-b616-4a9e-9dd9-ae7eefd3e847\") " pod="openshift-ingress/e2e-api-a-c24d61-openshift-default-7c56bd489-c85l8"
Oct 30 22:23:26.751969 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:26.751938    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd8986aa-b616-4a9e-9dd9-ae7eefd3e847-istio-token\") pod \"e2e-api-a-c24d61-openshift-default-7c56bd489-c85l8\" (UID: \"bd8986aa-b616-4a9e-9dd9-ae7eefd3e847\") " pod="openshift-ingress/e2e-api-a-c24d61-openshift-default-7c56bd489-c85l8"
Oct 30 22:23:26.752145 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:26.752127    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd8986aa-b616-4a9e-9dd9-ae7eefd3e847-credential-socket\") pod \"e2e-api-a-c24d61-openshift-default-7c56bd489-c85l8\" (UID: \"bd8986aa-b616-4a9e-9dd9-ae7eefd3e847\") " pod="openshift-ingress/e2e-api-a-c24d61-openshift-default-7c56bd489-c85l8"
Oct 30 22:23:26.752321 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:26.752286    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd8986aa-b616-4a9e-9dd9-ae7eefd3e847-istiod-ca-cert\") pod \"e2e-api-a-c24d61-openshift-default-7c56bd489-c85l8\" (UID: \"bd8986aa-b616-4a9e-9dd9-ae7eefd3e847\") " pod="openshift-ingress/e2e-api-a-c24d61-openshift-default-7c56bd489-c85l8"
Oct 30 22:23:26.756222 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:26.755663    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd8986aa-b616-4a9e-9dd9-ae7eefd3e847-istio-data\") pod \"e2e-api-a-c24d61-openshift-default-7c56bd489-c85l8\" (UID: \"bd8986aa-b616-4a9e-9dd9-ae7eefd3e847\") " pod="openshift-ingress/e2e-api-a-c24d61-openshift-default-7c56bd489-c85l8"
Oct 30 22:23:26.756222 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:26.755748    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd8986aa-b616-4a9e-9dd9-ae7eefd3e847-istio-envoy\") pod \"e2e-api-a-c24d61-openshift-default-7c56bd489-c85l8\" (UID: \"bd8986aa-b616-4a9e-9dd9-ae7eefd3e847\") " pod="openshift-ingress/e2e-api-a-c24d61-openshift-default-7c56bd489-c85l8"
Oct 30 22:23:26.756222 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:26.755842    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd8986aa-b616-4a9e-9dd9-ae7eefd3e847-istio-podinfo\") pod \"e2e-api-a-c24d61-openshift-default-7c56bd489-c85l8\" (UID: \"bd8986aa-b616-4a9e-9dd9-ae7eefd3e847\") " pod="openshift-ingress/e2e-api-a-c24d61-openshift-default-7c56bd489-c85l8"
Oct 30 22:23:26.756222 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:26.755892    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd8986aa-b616-4a9e-9dd9-ae7eefd3e847-workload-certs\") pod \"e2e-api-a-c24d61-openshift-default-7c56bd489-c85l8\" (UID: \"bd8986aa-b616-4a9e-9dd9-ae7eefd3e847\") " pod="openshift-ingress/e2e-api-a-c24d61-openshift-default-7c56bd489-c85l8"
Oct 30 22:23:26.756222 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:26.755958    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cp5tl\" (UniqueName: \"kubernetes.io/projected/bd8986aa-b616-4a9e-9dd9-ae7eefd3e847-kube-api-access-cp5tl\") pod \"e2e-api-a-c24d61-openshift-default-7c56bd489-c85l8\" (UID: \"bd8986aa-b616-4a9e-9dd9-ae7eefd3e847\") " pod="openshift-ingress/e2e-api-a-c24d61-openshift-default-7c56bd489-c85l8"
Oct 30 22:23:26.802649 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:26.802566    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-c24d61-openshift-default-7c56bd489-c85l8"]
Oct 30 22:23:26.858089 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:26.858027    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd8986aa-b616-4a9e-9dd9-ae7eefd3e847-istio-data\") pod \"e2e-api-a-c24d61-openshift-default-7c56bd489-c85l8\" (UID: \"bd8986aa-b616-4a9e-9dd9-ae7eefd3e847\") " pod="openshift-ingress/e2e-api-a-c24d61-openshift-default-7c56bd489-c85l8"
Oct 30 22:23:26.858290 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:26.858112    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd8986aa-b616-4a9e-9dd9-ae7eefd3e847-istio-envoy\") pod \"e2e-api-a-c24d61-openshift-default-7c56bd489-c85l8\" (UID: \"bd8986aa-b616-4a9e-9dd9-ae7eefd3e847\") " pod="openshift-ingress/e2e-api-a-c24d61-openshift-default-7c56bd489-c85l8"
Oct 30 22:23:26.858290 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:26.858196    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd8986aa-b616-4a9e-9dd9-ae7eefd3e847-istio-podinfo\") pod \"e2e-api-a-c24d61-openshift-default-7c56bd489-c85l8\" (UID: \"bd8986aa-b616-4a9e-9dd9-ae7eefd3e847\") " pod="openshift-ingress/e2e-api-a-c24d61-openshift-default-7c56bd489-c85l8"
Oct 30 22:23:26.858491 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:26.858292    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd8986aa-b616-4a9e-9dd9-ae7eefd3e847-workload-certs\") pod \"e2e-api-a-c24d61-openshift-default-7c56bd489-c85l8\" (UID: \"bd8986aa-b616-4a9e-9dd9-ae7eefd3e847\") " pod="openshift-ingress/e2e-api-a-c24d61-openshift-default-7c56bd489-c85l8"
Oct 30 22:23:26.858491 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:26.858382    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cp5tl\" (UniqueName: \"kubernetes.io/projected/bd8986aa-b616-4a9e-9dd9-ae7eefd3e847-kube-api-access-cp5tl\") pod \"e2e-api-a-c24d61-openshift-default-7c56bd489-c85l8\" (UID: \"bd8986aa-b616-4a9e-9dd9-ae7eefd3e847\") " pod="openshift-ingress/e2e-api-a-c24d61-openshift-default-7c56bd489-c85l8"
Oct 30 22:23:26.859055 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:26.858622    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd8986aa-b616-4a9e-9dd9-ae7eefd3e847-workload-socket\") pod \"e2e-api-a-c24d61-openshift-default-7c56bd489-c85l8\" (UID: \"bd8986aa-b616-4a9e-9dd9-ae7eefd3e847\") " pod="openshift-ingress/e2e-api-a-c24d61-openshift-default-7c56bd489-c85l8"
Oct 30 22:23:26.859055 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:26.858657    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd8986aa-b616-4a9e-9dd9-ae7eefd3e847-istio-token\") pod \"e2e-api-a-c24d61-openshift-default-7c56bd489-c85l8\" (UID: \"bd8986aa-b616-4a9e-9dd9-ae7eefd3e847\") " pod="openshift-ingress/e2e-api-a-c24d61-openshift-default-7c56bd489-c85l8"
Oct 30 22:23:26.859055 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:26.858685    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd8986aa-b616-4a9e-9dd9-ae7eefd3e847-credential-socket\") pod \"e2e-api-a-c24d61-openshift-default-7c56bd489-c85l8\" (UID: \"bd8986aa-b616-4a9e-9dd9-ae7eefd3e847\") " pod="openshift-ingress/e2e-api-a-c24d61-openshift-default-7c56bd489-c85l8"
Oct 30 22:23:26.859055 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:26.858728    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd8986aa-b616-4a9e-9dd9-ae7eefd3e847-istiod-ca-cert\") pod \"e2e-api-a-c24d61-openshift-default-7c56bd489-c85l8\" (UID: \"bd8986aa-b616-4a9e-9dd9-ae7eefd3e847\") " pod="openshift-ingress/e2e-api-a-c24d61-openshift-default-7c56bd489-c85l8"
Oct 30 22:23:26.859577 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:26.859546    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd8986aa-b616-4a9e-9dd9-ae7eefd3e847-istiod-ca-cert\") pod \"e2e-api-a-c24d61-openshift-default-7c56bd489-c85l8\" (UID: \"bd8986aa-b616-4a9e-9dd9-ae7eefd3e847\") " pod="openshift-ingress/e2e-api-a-c24d61-openshift-default-7c56bd489-c85l8"
Oct 30 22:23:26.861000 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:26.859946    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd8986aa-b616-4a9e-9dd9-ae7eefd3e847-workload-socket\") pod \"e2e-api-a-c24d61-openshift-default-7c56bd489-c85l8\" (UID: \"bd8986aa-b616-4a9e-9dd9-ae7eefd3e847\") " pod="openshift-ingress/e2e-api-a-c24d61-openshift-default-7c56bd489-c85l8"
Oct 30 22:23:26.861000 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:26.860204    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd8986aa-b616-4a9e-9dd9-ae7eefd3e847-workload-certs\") pod \"e2e-api-a-c24d61-openshift-default-7c56bd489-c85l8\" (UID: \"bd8986aa-b616-4a9e-9dd9-ae7eefd3e847\") " pod="openshift-ingress/e2e-api-a-c24d61-openshift-default-7c56bd489-c85l8"
Oct 30 22:23:26.861000 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:26.860941    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd8986aa-b616-4a9e-9dd9-ae7eefd3e847-credential-socket\") pod \"e2e-api-a-c24d61-openshift-default-7c56bd489-c85l8\" (UID: \"bd8986aa-b616-4a9e-9dd9-ae7eefd3e847\") " pod="openshift-ingress/e2e-api-a-c24d61-openshift-default-7c56bd489-c85l8"
Oct 30 22:23:26.861505 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:26.861485    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd8986aa-b616-4a9e-9dd9-ae7eefd3e847-istio-data\") pod \"e2e-api-a-c24d61-openshift-default-7c56bd489-c85l8\" (UID: \"bd8986aa-b616-4a9e-9dd9-ae7eefd3e847\") " pod="openshift-ingress/e2e-api-a-c24d61-openshift-default-7c56bd489-c85l8"
Oct 30 22:23:26.861745 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:26.861732    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd8986aa-b616-4a9e-9dd9-ae7eefd3e847-istio-envoy\") pod \"e2e-api-a-c24d61-openshift-default-7c56bd489-c85l8\" (UID: \"bd8986aa-b616-4a9e-9dd9-ae7eefd3e847\") " pod="openshift-ingress/e2e-api-a-c24d61-openshift-default-7c56bd489-c85l8"
Oct 30 22:23:26.865879 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:26.864187    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd8986aa-b616-4a9e-9dd9-ae7eefd3e847-istio-podinfo\") pod \"e2e-api-a-c24d61-openshift-default-7c56bd489-c85l8\" (UID: \"bd8986aa-b616-4a9e-9dd9-ae7eefd3e847\") " pod="openshift-ingress/e2e-api-a-c24d61-openshift-default-7c56bd489-c85l8"
Oct 30 22:23:26.919290 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:26.919261    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd8986aa-b616-4a9e-9dd9-ae7eefd3e847-istio-token\") pod \"e2e-api-a-c24d61-openshift-default-7c56bd489-c85l8\" (UID: \"bd8986aa-b616-4a9e-9dd9-ae7eefd3e847\") " pod="openshift-ingress/e2e-api-a-c24d61-openshift-default-7c56bd489-c85l8"
Oct 30 22:23:26.919522 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:26.919497    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cp5tl\" (UniqueName: \"kubernetes.io/projected/bd8986aa-b616-4a9e-9dd9-ae7eefd3e847-kube-api-access-cp5tl\") pod \"e2e-api-a-c24d61-openshift-default-7c56bd489-c85l8\" (UID: \"bd8986aa-b616-4a9e-9dd9-ae7eefd3e847\") " pod="openshift-ingress/e2e-api-a-c24d61-openshift-default-7c56bd489-c85l8"
Oct 30 22:23:27.052323 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:27.052280    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c24d61-openshift-default-7c56bd489-c85l8"
Oct 30 22:23:27.524665 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:27.524631    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-c24d61-openshift-default-7c56bd489-c85l8"]
Oct 30 22:23:27.530468 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:23:27.530443    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd8986aa_b616_4a9e_9dd9_ae7eefd3e847.slice/crio-7aacd52bae060401b4048f72cd22b8fd51bf26192fe905640071e0e0aab3fbe9 WatchSource:0}: Error finding container 7aacd52bae060401b4048f72cd22b8fd51bf26192fe905640071e0e0aab3fbe9: Status 404 returned error can't find the container with id 7aacd52bae060401b4048f72cd22b8fd51bf26192fe905640071e0e0aab3fbe9
Oct 30 22:23:27.533197 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:27.533163    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:23:27.533299 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:27.533233    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:23:27.533299 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:27.533265    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:23:27.752913 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:27.752881    5678 patch_prober.go:29] interesting pod/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-q828p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:23:27.753064 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:27.752938    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-q828p" podUID="4e8c9108-98ce-4a09-a8a9-f13def9228df" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:23:28.301911 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:28.301874    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-q828p"
Oct 30 22:23:28.340287 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:28.340253    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c24d61-openshift-default-7c56bd489-c85l8" event={"ID":"bd8986aa-b616-4a9e-9dd9-ae7eefd3e847","Type":"ContainerStarted","Data":"8bed24e326a634e2ce195cb92ff0d418033527731d0cb1f61ca976b2d01ce2af"}
Oct 30 22:23:28.340464 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:28.340349    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c24d61-openshift-default-7c56bd489-c85l8" event={"ID":"bd8986aa-b616-4a9e-9dd9-ae7eefd3e847","Type":"ContainerStarted","Data":"7aacd52bae060401b4048f72cd22b8fd51bf26192fe905640071e0e0aab3fbe9"}
Oct 30 22:23:28.342304 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:28.342269    5678 generic.go:356] "Generic (PLEG): container finished" podID="52992d7b-821d-4b13-945c-d0b9b59ddc58" containerID="baa9ce10b5014f02d6861aa75dfc8069cbb7fa6cf3d720e9bdd79818ffa1c287" exitCode=0
Oct 30 22:23:28.342457 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:28.342417    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-fpsll" event={"ID":"52992d7b-821d-4b13-945c-d0b9b59ddc58","Type":"ContainerDied","Data":"baa9ce10b5014f02d6861aa75dfc8069cbb7fa6cf3d720e9bdd79818ffa1c287"}
Oct 30 22:23:28.344544 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:28.344509    5678 generic.go:356] "Generic (PLEG): container finished" podID="0e17f2f4-e217-43cf-81ba-2bfb2095dbd6" containerID="21e5075edd24d61f4ec91b159a9a0ffa69bdae573523d0dc2eeff217c8a37caf" exitCode=0
Oct 30 22:23:28.345035 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:28.344632    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ldgt7" event={"ID":"0e17f2f4-e217-43cf-81ba-2bfb2095dbd6","Type":"ContainerDied","Data":"21e5075edd24d61f4ec91b159a9a0ffa69bdae573523d0dc2eeff217c8a37caf"}
Oct 30 22:23:28.346263 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:28.346241    5678 generic.go:356] "Generic (PLEG): container finished" podID="4e8c9108-98ce-4a09-a8a9-f13def9228df" containerID="6850f4ac14eb5dfdc02469456f3c2aaae042e5fe7ab85bbc8fa45a39e4bd3435" exitCode=0
Oct 30 22:23:28.346377 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:28.346292    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-q828p" event={"ID":"4e8c9108-98ce-4a09-a8a9-f13def9228df","Type":"ContainerDied","Data":"6850f4ac14eb5dfdc02469456f3c2aaae042e5fe7ab85bbc8fa45a39e4bd3435"}
Oct 30 22:23:28.346377 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:28.346328    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-q828p" event={"ID":"4e8c9108-98ce-4a09-a8a9-f13def9228df","Type":"ContainerDied","Data":"3ae2b279a90bcbbe7afe2914dcdd6456942e608b059cedea1280d60fb7a7a285"}
Oct 30 22:23:28.346377 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:28.346355    5678 scope.go:122] "RemoveContainer" containerID="6850f4ac14eb5dfdc02469456f3c2aaae042e5fe7ab85bbc8fa45a39e4bd3435"
Oct 30 22:23:28.346577 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:28.346562    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-q828p"
Oct 30 22:23:28.359665 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:28.359635    5678 scope.go:122] "RemoveContainer" containerID="6850f4ac14eb5dfdc02469456f3c2aaae042e5fe7ab85bbc8fa45a39e4bd3435"
Oct 30 22:23:28.360091 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:23:28.360066    5678 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6850f4ac14eb5dfdc02469456f3c2aaae042e5fe7ab85bbc8fa45a39e4bd3435\": container with ID starting with 6850f4ac14eb5dfdc02469456f3c2aaae042e5fe7ab85bbc8fa45a39e4bd3435 not found: ID does not exist" containerID="6850f4ac14eb5dfdc02469456f3c2aaae042e5fe7ab85bbc8fa45a39e4bd3435"
Oct 30 22:23:28.360226 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:28.360104    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6850f4ac14eb5dfdc02469456f3c2aaae042e5fe7ab85bbc8fa45a39e4bd3435"} err="failed to get container status \"6850f4ac14eb5dfdc02469456f3c2aaae042e5fe7ab85bbc8fa45a39e4bd3435\": rpc error: code = NotFound desc = could not find container \"6850f4ac14eb5dfdc02469456f3c2aaae042e5fe7ab85bbc8fa45a39e4bd3435\": container with ID starting with 6850f4ac14eb5dfdc02469456f3c2aaae042e5fe7ab85bbc8fa45a39e4bd3435 not found: ID does not exist"
Oct 30 22:23:28.367576 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:28.367557    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-fpsll"
Oct 30 22:23:28.383122 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:28.383095    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52992d7b-821d-4b13-945c-d0b9b59ddc58-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52992d7b-821d-4b13-945c-d0b9b59ddc58-workload-socket\") pod \"52992d7b-821d-4b13-945c-d0b9b59ddc58\" (UID: \"52992d7b-821d-4b13-945c-d0b9b59ddc58\") "
Oct 30 22:23:28.383236 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:28.383198    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e8c9108-98ce-4a09-a8a9-f13def9228df-kube-api-access-ndlj2\" (UniqueName: \"kubernetes.io/projected/4e8c9108-98ce-4a09-a8a9-f13def9228df-kube-api-access-ndlj2\") pod \"4e8c9108-98ce-4a09-a8a9-f13def9228df\" (UID: \"4e8c9108-98ce-4a09-a8a9-f13def9228df\") "
Oct 30 22:23:28.383301 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:28.383233    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52992d7b-821d-4b13-945c-d0b9b59ddc58-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52992d7b-821d-4b13-945c-d0b9b59ddc58-istiod-ca-cert\") pod \"52992d7b-821d-4b13-945c-d0b9b59ddc58\" (UID: \"52992d7b-821d-4b13-945c-d0b9b59ddc58\") "
Oct 30 22:23:28.383301 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:28.383260    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4e8c9108-98ce-4a09-a8a9-f13def9228df-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e8c9108-98ce-4a09-a8a9-f13def9228df-istio-podinfo\") pod \"4e8c9108-98ce-4a09-a8a9-f13def9228df\" (UID: \"4e8c9108-98ce-4a09-a8a9-f13def9228df\") "
Oct 30 22:23:28.383301 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:28.383295    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52992d7b-821d-4b13-945c-d0b9b59ddc58-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52992d7b-821d-4b13-945c-d0b9b59ddc58-istio-envoy\") pod \"52992d7b-821d-4b13-945c-d0b9b59ddc58\" (UID: \"52992d7b-821d-4b13-945c-d0b9b59ddc58\") "
Oct 30 22:23:28.383484 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:28.383325    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e8c9108-98ce-4a09-a8a9-f13def9228df-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e8c9108-98ce-4a09-a8a9-f13def9228df-istio-data\") pod \"4e8c9108-98ce-4a09-a8a9-f13def9228df\" (UID: \"4e8c9108-98ce-4a09-a8a9-f13def9228df\") "
Oct 30 22:23:28.383484 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:28.383329    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52992d7b-821d-4b13-945c-d0b9b59ddc58-workload-socket" pod "52992d7b-821d-4b13-945c-d0b9b59ddc58" (UID: "52992d7b-821d-4b13-945c-d0b9b59ddc58"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:23:28.383484 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:28.383467    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52992d7b-821d-4b13-945c-d0b9b59ddc58-kube-api-access-rf2s2\" (UniqueName: \"kubernetes.io/projected/52992d7b-821d-4b13-945c-d0b9b59ddc58-kube-api-access-rf2s2\") pod \"52992d7b-821d-4b13-945c-d0b9b59ddc58\" (UID: \"52992d7b-821d-4b13-945c-d0b9b59ddc58\") "
Oct 30 22:23:28.383655 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:28.383568    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52992d7b-821d-4b13-945c-d0b9b59ddc58-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52992d7b-821d-4b13-945c-d0b9b59ddc58-credential-socket\") pod \"52992d7b-821d-4b13-945c-d0b9b59ddc58\" (UID: \"52992d7b-821d-4b13-945c-d0b9b59ddc58\") "
Oct 30 22:23:28.383655 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:28.383630    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52992d7b-821d-4b13-945c-d0b9b59ddc58-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52992d7b-821d-4b13-945c-d0b9b59ddc58-workload-certs\") pod \"52992d7b-821d-4b13-945c-d0b9b59ddc58\" (UID: \"52992d7b-821d-4b13-945c-d0b9b59ddc58\") "
Oct 30 22:23:28.383765 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:28.383661    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e8c9108-98ce-4a09-a8a9-f13def9228df-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e8c9108-98ce-4a09-a8a9-f13def9228df-credential-socket\") pod \"4e8c9108-98ce-4a09-a8a9-f13def9228df\" (UID: \"4e8c9108-98ce-4a09-a8a9-f13def9228df\") "
Oct 30 22:23:28.383765 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:28.383737    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52992d7b-821d-4b13-945c-d0b9b59ddc58-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52992d7b-821d-4b13-945c-d0b9b59ddc58-istio-data\") pod \"52992d7b-821d-4b13-945c-d0b9b59ddc58\" (UID: \"52992d7b-821d-4b13-945c-d0b9b59ddc58\") "
Oct 30 22:23:28.383935 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:28.383766    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e8c9108-98ce-4a09-a8a9-f13def9228df-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e8c9108-98ce-4a09-a8a9-f13def9228df-istio-envoy\") pod \"4e8c9108-98ce-4a09-a8a9-f13def9228df\" (UID: \"4e8c9108-98ce-4a09-a8a9-f13def9228df\") "
Oct 30 22:23:28.383935 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:28.383798    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e8c9108-98ce-4a09-a8a9-f13def9228df-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e8c9108-98ce-4a09-a8a9-f13def9228df-workload-socket\") pod \"4e8c9108-98ce-4a09-a8a9-f13def9228df\" (UID: \"4e8c9108-98ce-4a09-a8a9-f13def9228df\") "
Oct 30 22:23:28.383935 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:28.383836    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e8c9108-98ce-4a09-a8a9-f13def9228df-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e8c9108-98ce-4a09-a8a9-f13def9228df-workload-certs\") pod \"4e8c9108-98ce-4a09-a8a9-f13def9228df\" (UID: \"4e8c9108-98ce-4a09-a8a9-f13def9228df\") "
Oct 30 22:23:28.383935 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:28.383866    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e8c9108-98ce-4a09-a8a9-f13def9228df-istio-token\" (UniqueName: \"kubernetes.io/projected/4e8c9108-98ce-4a09-a8a9-f13def9228df-istio-token\") pod \"4e8c9108-98ce-4a09-a8a9-f13def9228df\" (UID: \"4e8c9108-98ce-4a09-a8a9-f13def9228df\") "
Oct 30 22:23:28.383935 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:28.383918    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52992d7b-821d-4b13-945c-d0b9b59ddc58-istiod-ca-cert" pod "52992d7b-821d-4b13-945c-d0b9b59ddc58" (UID: "52992d7b-821d-4b13-945c-d0b9b59ddc58"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:23:28.384254 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:28.383944    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52992d7b-821d-4b13-945c-d0b9b59ddc58-istio-token\" (UniqueName: \"kubernetes.io/projected/52992d7b-821d-4b13-945c-d0b9b59ddc58-istio-token\") pod \"52992d7b-821d-4b13-945c-d0b9b59ddc58\" (UID: \"52992d7b-821d-4b13-945c-d0b9b59ddc58\") "
Oct 30 22:23:28.384254 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:28.383979    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/52992d7b-821d-4b13-945c-d0b9b59ddc58-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52992d7b-821d-4b13-945c-d0b9b59ddc58-istio-podinfo\") pod \"52992d7b-821d-4b13-945c-d0b9b59ddc58\" (UID: \"52992d7b-821d-4b13-945c-d0b9b59ddc58\") "
Oct 30 22:23:28.384254 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:28.384013    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e8c9108-98ce-4a09-a8a9-f13def9228df-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e8c9108-98ce-4a09-a8a9-f13def9228df-istiod-ca-cert\") pod \"4e8c9108-98ce-4a09-a8a9-f13def9228df\" (UID: \"4e8c9108-98ce-4a09-a8a9-f13def9228df\") "
Oct 30 22:23:28.384254 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:28.384144    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52992d7b-821d-4b13-945c-d0b9b59ddc58-credential-socket" pod "52992d7b-821d-4b13-945c-d0b9b59ddc58" (UID: "52992d7b-821d-4b13-945c-d0b9b59ddc58"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:23:28.384497 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:28.384473    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e8c9108-98ce-4a09-a8a9-f13def9228df-istio-data" pod "4e8c9108-98ce-4a09-a8a9-f13def9228df" (UID: "4e8c9108-98ce-4a09-a8a9-f13def9228df"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:23:28.384913 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:28.384889    5678 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52992d7b-821d-4b13-945c-d0b9b59ddc58-workload-socket\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:28.385002 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:28.384919    5678 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52992d7b-821d-4b13-945c-d0b9b59ddc58-istiod-ca-cert\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:28.385002 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:28.384934    5678 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e8c9108-98ce-4a09-a8a9-f13def9228df-istio-data\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:28.385002 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:28.384947    5678 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52992d7b-821d-4b13-945c-d0b9b59ddc58-credential-socket\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:28.385546 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:28.385520    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e8c9108-98ce-4a09-a8a9-f13def9228df-istiod-ca-cert" pod "4e8c9108-98ce-4a09-a8a9-f13def9228df" (UID: "4e8c9108-98ce-4a09-a8a9-f13def9228df"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:23:28.385945 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:28.385921    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e8c9108-98ce-4a09-a8a9-f13def9228df-kube-api-access-ndlj2" pod "4e8c9108-98ce-4a09-a8a9-f13def9228df" (UID: "4e8c9108-98ce-4a09-a8a9-f13def9228df"). InnerVolumeSpecName "kube-api-access-ndlj2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:23:28.386597 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:28.386572    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e8c9108-98ce-4a09-a8a9-f13def9228df-credential-socket" pod "4e8c9108-98ce-4a09-a8a9-f13def9228df" (UID: "4e8c9108-98ce-4a09-a8a9-f13def9228df"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:23:28.386790 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:28.386766    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52992d7b-821d-4b13-945c-d0b9b59ddc58-workload-certs" pod "52992d7b-821d-4b13-945c-d0b9b59ddc58" (UID: "52992d7b-821d-4b13-945c-d0b9b59ddc58"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:23:28.386956 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:28.386931    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e8c9108-98ce-4a09-a8a9-f13def9228df-workload-socket" pod "4e8c9108-98ce-4a09-a8a9-f13def9228df" (UID: "4e8c9108-98ce-4a09-a8a9-f13def9228df"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:23:28.387044 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:28.387004    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4e8c9108-98ce-4a09-a8a9-f13def9228df-istio-podinfo" pod "4e8c9108-98ce-4a09-a8a9-f13def9228df" (UID: "4e8c9108-98ce-4a09-a8a9-f13def9228df"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:23:28.387122 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:28.387105    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e8c9108-98ce-4a09-a8a9-f13def9228df-workload-certs" pod "4e8c9108-98ce-4a09-a8a9-f13def9228df" (UID: "4e8c9108-98ce-4a09-a8a9-f13def9228df"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:23:28.387234 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:28.387217    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52992d7b-821d-4b13-945c-d0b9b59ddc58-istio-data" pod "52992d7b-821d-4b13-945c-d0b9b59ddc58" (UID: "52992d7b-821d-4b13-945c-d0b9b59ddc58"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:23:28.387739 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:28.387709    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52992d7b-821d-4b13-945c-d0b9b59ddc58-istio-envoy" pod "52992d7b-821d-4b13-945c-d0b9b59ddc58" (UID: "52992d7b-821d-4b13-945c-d0b9b59ddc58"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:23:28.387841 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:28.387816    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/52992d7b-821d-4b13-945c-d0b9b59ddc58-istio-podinfo" pod "52992d7b-821d-4b13-945c-d0b9b59ddc58" (UID: "52992d7b-821d-4b13-945c-d0b9b59ddc58"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:23:28.387841 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:28.387815    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e8c9108-98ce-4a09-a8a9-f13def9228df-istio-envoy" pod "4e8c9108-98ce-4a09-a8a9-f13def9228df" (UID: "4e8c9108-98ce-4a09-a8a9-f13def9228df"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:23:28.387930 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:28.387869    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e8c9108-98ce-4a09-a8a9-f13def9228df-istio-token" pod "4e8c9108-98ce-4a09-a8a9-f13def9228df" (UID: "4e8c9108-98ce-4a09-a8a9-f13def9228df"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:23:28.387930 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:28.387898    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52992d7b-821d-4b13-945c-d0b9b59ddc58-kube-api-access-rf2s2" pod "52992d7b-821d-4b13-945c-d0b9b59ddc58" (UID: "52992d7b-821d-4b13-945c-d0b9b59ddc58"). InnerVolumeSpecName "kube-api-access-rf2s2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:23:28.388492 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:28.388470    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52992d7b-821d-4b13-945c-d0b9b59ddc58-istio-token" pod "52992d7b-821d-4b13-945c-d0b9b59ddc58" (UID: "52992d7b-821d-4b13-945c-d0b9b59ddc58"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:23:28.418444 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:28.418422    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ldgt7"
Oct 30 22:23:28.486148 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:28.486116    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0e17f2f4-e217-43cf-81ba-2bfb2095dbd6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e17f2f4-e217-43cf-81ba-2bfb2095dbd6-istio-podinfo\") pod \"0e17f2f4-e217-43cf-81ba-2bfb2095dbd6\" (UID: \"0e17f2f4-e217-43cf-81ba-2bfb2095dbd6\") "
Oct 30 22:23:28.486335 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:28.486200    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e17f2f4-e217-43cf-81ba-2bfb2095dbd6-istio-token\" (UniqueName: \"kubernetes.io/projected/0e17f2f4-e217-43cf-81ba-2bfb2095dbd6-istio-token\") pod \"0e17f2f4-e217-43cf-81ba-2bfb2095dbd6\" (UID: \"0e17f2f4-e217-43cf-81ba-2bfb2095dbd6\") "
Oct 30 22:23:28.486335 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:28.486282    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e17f2f4-e217-43cf-81ba-2bfb2095dbd6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e17f2f4-e217-43cf-81ba-2bfb2095dbd6-credential-socket\") pod \"0e17f2f4-e217-43cf-81ba-2bfb2095dbd6\" (UID: \"0e17f2f4-e217-43cf-81ba-2bfb2095dbd6\") "
Oct 30 22:23:28.486485 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:28.486333    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e17f2f4-e217-43cf-81ba-2bfb2095dbd6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e17f2f4-e217-43cf-81ba-2bfb2095dbd6-workload-certs\") pod \"0e17f2f4-e217-43cf-81ba-2bfb2095dbd6\" (UID: \"0e17f2f4-e217-43cf-81ba-2bfb2095dbd6\") "
Oct 30 22:23:28.486485 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:28.486444    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e17f2f4-e217-43cf-81ba-2bfb2095dbd6-kube-api-access-7dr7d\" (UniqueName: \"kubernetes.io/projected/0e17f2f4-e217-43cf-81ba-2bfb2095dbd6-kube-api-access-7dr7d\") pod \"0e17f2f4-e217-43cf-81ba-2bfb2095dbd6\" (UID: \"0e17f2f4-e217-43cf-81ba-2bfb2095dbd6\") "
Oct 30 22:23:28.486591 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:28.486480    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e17f2f4-e217-43cf-81ba-2bfb2095dbd6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e17f2f4-e217-43cf-81ba-2bfb2095dbd6-istio-envoy\") pod \"0e17f2f4-e217-43cf-81ba-2bfb2095dbd6\" (UID: \"0e17f2f4-e217-43cf-81ba-2bfb2095dbd6\") "
Oct 30 22:23:28.486591 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:28.486524    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e17f2f4-e217-43cf-81ba-2bfb2095dbd6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e17f2f4-e217-43cf-81ba-2bfb2095dbd6-workload-socket\") pod \"0e17f2f4-e217-43cf-81ba-2bfb2095dbd6\" (UID: \"0e17f2f4-e217-43cf-81ba-2bfb2095dbd6\") "
Oct 30 22:23:28.486591 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:28.486558    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e17f2f4-e217-43cf-81ba-2bfb2095dbd6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e17f2f4-e217-43cf-81ba-2bfb2095dbd6-istiod-ca-cert\") pod \"0e17f2f4-e217-43cf-81ba-2bfb2095dbd6\" (UID: \"0e17f2f4-e217-43cf-81ba-2bfb2095dbd6\") "
Oct 30 22:23:28.486750 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:28.486680    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e17f2f4-e217-43cf-81ba-2bfb2095dbd6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e17f2f4-e217-43cf-81ba-2bfb2095dbd6-istio-data\") pod \"0e17f2f4-e217-43cf-81ba-2bfb2095dbd6\" (UID: \"0e17f2f4-e217-43cf-81ba-2bfb2095dbd6\") "
Oct 30 22:23:28.487591 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:28.487002    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e17f2f4-e217-43cf-81ba-2bfb2095dbd6-workload-socket" pod "0e17f2f4-e217-43cf-81ba-2bfb2095dbd6" (UID: "0e17f2f4-e217-43cf-81ba-2bfb2095dbd6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:23:28.487591 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:28.487019    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e17f2f4-e217-43cf-81ba-2bfb2095dbd6-workload-certs" pod "0e17f2f4-e217-43cf-81ba-2bfb2095dbd6" (UID: "0e17f2f4-e217-43cf-81ba-2bfb2095dbd6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:23:28.487591 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:28.487199    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e17f2f4-e217-43cf-81ba-2bfb2095dbd6-istio-data" pod "0e17f2f4-e217-43cf-81ba-2bfb2095dbd6" (UID: "0e17f2f4-e217-43cf-81ba-2bfb2095dbd6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:23:28.487591 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:28.487383    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e17f2f4-e217-43cf-81ba-2bfb2095dbd6-credential-socket" pod "0e17f2f4-e217-43cf-81ba-2bfb2095dbd6" (UID: "0e17f2f4-e217-43cf-81ba-2bfb2095dbd6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:23:28.487591 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:28.487447    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e17f2f4-e217-43cf-81ba-2bfb2095dbd6-istiod-ca-cert" pod "0e17f2f4-e217-43cf-81ba-2bfb2095dbd6" (UID: "0e17f2f4-e217-43cf-81ba-2bfb2095dbd6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:23:28.487591 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:28.487546    5678 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52992d7b-821d-4b13-945c-d0b9b59ddc58-workload-certs\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:28.487591 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:28.487569    5678 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e8c9108-98ce-4a09-a8a9-f13def9228df-credential-socket\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:28.487591 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:28.487582    5678 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52992d7b-821d-4b13-945c-d0b9b59ddc58-istio-data\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:28.487591 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:28.487596    5678 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e8c9108-98ce-4a09-a8a9-f13def9228df-istio-envoy\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:28.488161 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:28.487610    5678 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e8c9108-98ce-4a09-a8a9-f13def9228df-workload-socket\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:28.488161 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:28.487624    5678 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e8c9108-98ce-4a09-a8a9-f13def9228df-workload-certs\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:28.488161 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:28.487638    5678 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e8c9108-98ce-4a09-a8a9-f13def9228df-istio-token\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:28.488161 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:28.487651    5678 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52992d7b-821d-4b13-945c-d0b9b59ddc58-istio-token\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:28.488161 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:28.487664    5678 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52992d7b-821d-4b13-945c-d0b9b59ddc58-istio-podinfo\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:28.488161 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:28.487678    5678 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e8c9108-98ce-4a09-a8a9-f13def9228df-istiod-ca-cert\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:28.488161 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:28.487690    5678 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e17f2f4-e217-43cf-81ba-2bfb2095dbd6-credential-socket\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:28.488161 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:28.487704    5678 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e17f2f4-e217-43cf-81ba-2bfb2095dbd6-workload-certs\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:28.488161 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:28.487718    5678 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e17f2f4-e217-43cf-81ba-2bfb2095dbd6-workload-socket\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:28.488161 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:28.487733    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndlj2\" (UniqueName: \"kubernetes.io/projected/4e8c9108-98ce-4a09-a8a9-f13def9228df-kube-api-access-ndlj2\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:28.488161 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:28.487749    5678 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e8c9108-98ce-4a09-a8a9-f13def9228df-istio-podinfo\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:28.488161 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:28.487762    5678 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52992d7b-821d-4b13-945c-d0b9b59ddc58-istio-envoy\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:28.488161 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:28.487776    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rf2s2\" (UniqueName: \"kubernetes.io/projected/52992d7b-821d-4b13-945c-d0b9b59ddc58-kube-api-access-rf2s2\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:28.488161 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:28.487790    5678 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e17f2f4-e217-43cf-81ba-2bfb2095dbd6-istio-data\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:28.489086 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:28.489056    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e17f2f4-e217-43cf-81ba-2bfb2095dbd6-istio-token" pod "0e17f2f4-e217-43cf-81ba-2bfb2095dbd6" (UID: "0e17f2f4-e217-43cf-81ba-2bfb2095dbd6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:23:28.489780 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:28.489755    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e17f2f4-e217-43cf-81ba-2bfb2095dbd6-istio-envoy" pod "0e17f2f4-e217-43cf-81ba-2bfb2095dbd6" (UID: "0e17f2f4-e217-43cf-81ba-2bfb2095dbd6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:23:28.495152 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:28.495119    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e17f2f4-e217-43cf-81ba-2bfb2095dbd6-kube-api-access-7dr7d" pod "0e17f2f4-e217-43cf-81ba-2bfb2095dbd6" (UID: "0e17f2f4-e217-43cf-81ba-2bfb2095dbd6"). InnerVolumeSpecName "kube-api-access-7dr7d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:23:28.496627 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:28.496602    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0e17f2f4-e217-43cf-81ba-2bfb2095dbd6-istio-podinfo" pod "0e17f2f4-e217-43cf-81ba-2bfb2095dbd6" (UID: "0e17f2f4-e217-43cf-81ba-2bfb2095dbd6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:23:28.589246 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:28.589216    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dr7d\" (UniqueName: \"kubernetes.io/projected/0e17f2f4-e217-43cf-81ba-2bfb2095dbd6-kube-api-access-7dr7d\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:28.589246 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:28.589246    5678 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e17f2f4-e217-43cf-81ba-2bfb2095dbd6-istio-envoy\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:28.589700 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:28.589256    5678 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e17f2f4-e217-43cf-81ba-2bfb2095dbd6-istiod-ca-cert\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:28.589700 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:28.589264    5678 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e17f2f4-e217-43cf-81ba-2bfb2095dbd6-istio-podinfo\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:28.589700 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:28.589272    5678 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e17f2f4-e217-43cf-81ba-2bfb2095dbd6-istio-token\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:28.657552 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:28.657491    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-c24d61-openshift-default-7c56bd489-c85l8" podStartSLOduration=2.657473019 podStartE2EDuration="2.657473019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:23:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:23:28.656538355 +0000 UTC m=+1986.260845909" watchObservedRunningTime="2026-10-30 22:23:28.657473019 +0000 UTC m=+1986.261780572"
Oct 30 22:23:28.947553 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:28.947521    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-q828p"]
Oct 30 22:23:28.951431 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:28.951373    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-q828p"]
Oct 30 22:23:29.053093 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:29.053063    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-c24d61-openshift-default-7c56bd489-c85l8"
Oct 30 22:23:29.059643 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:29.059618    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-c24d61-openshift-default-7c56bd489-c85l8"
Oct 30 22:23:29.357374 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:29.357288    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-fpsll"
Oct 30 22:23:29.357374 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:29.357308    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-fpsll" event={"ID":"52992d7b-821d-4b13-945c-d0b9b59ddc58","Type":"ContainerDied","Data":"179fb133649751bb685ace2e0c7ede726009431642a8d86372197ab8c826cdf2"}
Oct 30 22:23:29.357374 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:29.357372    5678 scope.go:122] "RemoveContainer" containerID="baa9ce10b5014f02d6861aa75dfc8069cbb7fa6cf3d720e9bdd79818ffa1c287"
Oct 30 22:23:29.358920 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:29.358901    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ldgt7"
Oct 30 22:23:29.359037 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:29.358934    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ldgt7" event={"ID":"0e17f2f4-e217-43cf-81ba-2bfb2095dbd6","Type":"ContainerDied","Data":"bbf89821c56c69ae802e32748e2dae32b3c7ef3c53f6187fb9948aa057138542"}
Oct 30 22:23:29.360511 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:29.360456    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-c24d61-openshift-default-7c56bd489-c85l8"
Oct 30 22:23:29.361680 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:29.361657    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-c24d61-openshift-default-7c56bd489-c85l8"
Oct 30 22:23:29.377486 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:29.377459    5678 scope.go:122] "RemoveContainer" containerID="21e5075edd24d61f4ec91b159a9a0ffa69bdae573523d0dc2eeff217c8a37caf"
Oct 30 22:23:29.518798 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:29.518766    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ldgt7"]
Oct 30 22:23:29.523063 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:29.523027    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ldgt7"]
Oct 30 22:23:29.759311 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:29.759277    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-fpsll"]
Oct 30 22:23:29.762699 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:29.762676    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-fpsll"]
Oct 30 22:23:30.544836 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:30.544799    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e17f2f4-e217-43cf-81ba-2bfb2095dbd6" path="/var/lib/kubelet/pods/0e17f2f4-e217-43cf-81ba-2bfb2095dbd6/volumes"
Oct 30 22:23:30.545556 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:30.545511    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e8c9108-98ce-4a09-a8a9-f13def9228df" path="/var/lib/kubelet/pods/4e8c9108-98ce-4a09-a8a9-f13def9228df/volumes"
Oct 30 22:23:30.546195 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:30.546121    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52992d7b-821d-4b13-945c-d0b9b59ddc58" path="/var/lib/kubelet/pods/52992d7b-821d-4b13-945c-d0b9b59ddc58/volumes"
Oct 30 22:23:35.181133 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:35.181042    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c24d61-5b86d448c-69ql5"]
Oct 30 22:23:35.182045 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:35.182022    5678 cpu_manager.go:414] "containerMap: removing container" podUID="4e8c9108-98ce-4a09-a8a9-f13def9228df" containerName="istio-proxy"
Oct 30 22:23:35.182045 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:35.182049    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e8c9108-98ce-4a09-a8a9-f13def9228df" containerName="istio-proxy"
Oct 30 22:23:35.182154 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:35.182068    5678 cpu_manager.go:414] "containerMap: removing container" podUID="0e17f2f4-e217-43cf-81ba-2bfb2095dbd6" containerName="istio-proxy"
Oct 30 22:23:35.182154 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:35.182077    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e17f2f4-e217-43cf-81ba-2bfb2095dbd6" containerName="istio-proxy"
Oct 30 22:23:35.182154 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:35.182117    5678 cpu_manager.go:414] "containerMap: removing container" podUID="52992d7b-821d-4b13-945c-d0b9b59ddc58" containerName="istio-proxy"
Oct 30 22:23:35.182154 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:35.182125    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52992d7b-821d-4b13-945c-d0b9b59ddc58" containerName="istio-proxy"
Oct 30 22:23:35.182275 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:35.182261    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="52992d7b-821d-4b13-945c-d0b9b59ddc58" containerName="istio-proxy"
Oct 30 22:23:35.182315 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:35.182280    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e8c9108-98ce-4a09-a8a9-f13def9228df" containerName="istio-proxy"
Oct 30 22:23:35.182315 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:35.182292    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e17f2f4-e217-43cf-81ba-2bfb2095dbd6" containerName="istio-proxy"
Oct 30 22:23:35.189454 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:35.188119    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c24d61-5b86d448c-69ql5"
Oct 30 22:23:35.245727 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:35.245692    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c24d61-5b86d448c-69ql5"]
Oct 30 22:23:35.310266 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:35.310232    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-c24d61-dockercfg-zx8ss\""
Oct 30 22:23:35.310501 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:35.310473    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-c24d61\""
Oct 30 22:23:35.314844 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:35.314813    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cw588\" (UniqueName: \"kubernetes.io/projected/cafd6d42-da8f-4b65-bcfa-41e67dc3b90e-kube-api-access-cw588\") pod \"payload-pre-processing-e2e-api-a-c24d61-5b86d448c-69ql5\" (UID: \"cafd6d42-da8f-4b65-bcfa-41e67dc3b90e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c24d61-5b86d448c-69ql5"
Oct 30 22:23:35.315084 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:35.315066    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cafd6d42-da8f-4b65-bcfa-41e67dc3b90e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-c24d61-5b86d448c-69ql5\" (UID: \"cafd6d42-da8f-4b65-bcfa-41e67dc3b90e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c24d61-5b86d448c-69ql5"
Oct 30 22:23:35.317258 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:35.317230    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c24d61-5b9ccff96c-klgks"]
Oct 30 22:23:35.322682 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:35.322659    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c24d61-5b9ccff96c-klgks"
Oct 30 22:23:35.374072 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:35.374043    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c24d61-5b9ccff96c-klgks"]
Oct 30 22:23:35.419751 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:35.419706    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cw588\" (UniqueName: \"kubernetes.io/projected/cafd6d42-da8f-4b65-bcfa-41e67dc3b90e-kube-api-access-cw588\") pod \"payload-pre-processing-e2e-api-a-c24d61-5b86d448c-69ql5\" (UID: \"cafd6d42-da8f-4b65-bcfa-41e67dc3b90e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c24d61-5b86d448c-69ql5"
Oct 30 22:23:35.419952 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:35.419790    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cafd6d42-da8f-4b65-bcfa-41e67dc3b90e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-c24d61-5b86d448c-69ql5\" (UID: \"cafd6d42-da8f-4b65-bcfa-41e67dc3b90e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c24d61-5b86d448c-69ql5"
Oct 30 22:23:35.420924 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:35.420843    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cafd6d42-da8f-4b65-bcfa-41e67dc3b90e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-c24d61-5b86d448c-69ql5\" (UID: \"cafd6d42-da8f-4b65-bcfa-41e67dc3b90e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c24d61-5b86d448c-69ql5"
Oct 30 22:23:35.507516 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:35.501948    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cw588\" (UniqueName: \"kubernetes.io/projected/cafd6d42-da8f-4b65-bcfa-41e67dc3b90e-kube-api-access-cw588\") pod \"payload-pre-processing-e2e-api-a-c24d61-5b86d448c-69ql5\" (UID: \"cafd6d42-da8f-4b65-bcfa-41e67dc3b90e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c24d61-5b86d448c-69ql5"
Oct 30 22:23:35.517734 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:35.517701    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c24d61-5b86d448c-69ql5"
Oct 30 22:23:35.521601 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:35.520905    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7g92w\" (UniqueName: \"kubernetes.io/projected/9c1a81bf-5ffe-4f8a-9295-781400c2f261-kube-api-access-7g92w\") pod \"payload-processing-e2e-api-a-c24d61-5b9ccff96c-klgks\" (UID: \"9c1a81bf-5ffe-4f8a-9295-781400c2f261\") " pod="openshift-ingress/payload-processing-e2e-api-a-c24d61-5b9ccff96c-klgks"
Oct 30 22:23:35.521601 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:35.521065    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c1a81bf-5ffe-4f8a-9295-781400c2f261-plugins-config-volume\") pod \"payload-processing-e2e-api-a-c24d61-5b9ccff96c-klgks\" (UID: \"9c1a81bf-5ffe-4f8a-9295-781400c2f261\") " pod="openshift-ingress/payload-processing-e2e-api-a-c24d61-5b9ccff96c-klgks"
Oct 30 22:23:35.622019 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:35.621983    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7g92w\" (UniqueName: \"kubernetes.io/projected/9c1a81bf-5ffe-4f8a-9295-781400c2f261-kube-api-access-7g92w\") pod \"payload-processing-e2e-api-a-c24d61-5b9ccff96c-klgks\" (UID: \"9c1a81bf-5ffe-4f8a-9295-781400c2f261\") " pod="openshift-ingress/payload-processing-e2e-api-a-c24d61-5b9ccff96c-klgks"
Oct 30 22:23:35.622147 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:35.622122    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c1a81bf-5ffe-4f8a-9295-781400c2f261-plugins-config-volume\") pod \"payload-processing-e2e-api-a-c24d61-5b9ccff96c-klgks\" (UID: \"9c1a81bf-5ffe-4f8a-9295-781400c2f261\") " pod="openshift-ingress/payload-processing-e2e-api-a-c24d61-5b9ccff96c-klgks"
Oct 30 22:23:35.622813 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:35.622795    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c1a81bf-5ffe-4f8a-9295-781400c2f261-plugins-config-volume\") pod \"payload-processing-e2e-api-a-c24d61-5b9ccff96c-klgks\" (UID: \"9c1a81bf-5ffe-4f8a-9295-781400c2f261\") " pod="openshift-ingress/payload-processing-e2e-api-a-c24d61-5b9ccff96c-klgks"
Oct 30 22:23:35.681937 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:35.681911    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7g92w\" (UniqueName: \"kubernetes.io/projected/9c1a81bf-5ffe-4f8a-9295-781400c2f261-kube-api-access-7g92w\") pod \"payload-processing-e2e-api-a-c24d61-5b9ccff96c-klgks\" (UID: \"9c1a81bf-5ffe-4f8a-9295-781400c2f261\") " pod="openshift-ingress/payload-processing-e2e-api-a-c24d61-5b9ccff96c-klgks"
Oct 30 22:23:35.916989 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:35.916954    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c24d61-5b86d448c-69ql5"]
Oct 30 22:23:35.922876 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:23:35.922847    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcafd6d42_da8f_4b65_bcfa_41e67dc3b90e.slice/crio-bf30c433d9d164c313d6b69f7c7747a555030315d633e78b42e8817052ea812d WatchSource:0}: Error finding container bf30c433d9d164c313d6b69f7c7747a555030315d633e78b42e8817052ea812d: Status 404 returned error can't find the container with id bf30c433d9d164c313d6b69f7c7747a555030315d633e78b42e8817052ea812d
Oct 30 22:23:35.946101 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:35.946078    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c24d61-5b9ccff96c-klgks"
Oct 30 22:23:36.335011 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:36.334920    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c24d61-5b9ccff96c-klgks"]
Oct 30 22:23:36.341248 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:23:36.341208    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c1a81bf_5ffe_4f8a_9295_781400c2f261.slice/crio-1f668ee27f95a34e851d8f27bb82cf4a9c809c5a6522e19c320cbcfd20af1e70 WatchSource:0}: Error finding container 1f668ee27f95a34e851d8f27bb82cf4a9c809c5a6522e19c320cbcfd20af1e70: Status 404 returned error can't find the container with id 1f668ee27f95a34e851d8f27bb82cf4a9c809c5a6522e19c320cbcfd20af1e70
Oct 30 22:23:36.451317 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:36.451287    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c24d61-5b86d448c-69ql5" event={"ID":"cafd6d42-da8f-4b65-bcfa-41e67dc3b90e","Type":"ContainerStarted","Data":"e4cd15fd1503db611a73f1d3f1cefe2cd71eb4255e12f57a37f9e6b9a096fa31"}
Oct 30 22:23:36.451466 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:36.451347    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c24d61-5b86d448c-69ql5" event={"ID":"cafd6d42-da8f-4b65-bcfa-41e67dc3b90e","Type":"ContainerStarted","Data":"bf30c433d9d164c313d6b69f7c7747a555030315d633e78b42e8817052ea812d"}
Oct 30 22:23:36.451466 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:36.451373    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c24d61-5b86d448c-69ql5"
Oct 30 22:23:36.453133 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:36.453109    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c24d61-5b9ccff96c-klgks" event={"ID":"9c1a81bf-5ffe-4f8a-9295-781400c2f261","Type":"ContainerStarted","Data":"88f74be73331ce950de36b8cb82a5d458d93e0aa42bd9155394972ff9c2c8859"}
Oct 30 22:23:36.453246 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:36.453148    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c24d61-5b9ccff96c-klgks" event={"ID":"9c1a81bf-5ffe-4f8a-9295-781400c2f261","Type":"ContainerStarted","Data":"1f668ee27f95a34e851d8f27bb82cf4a9c809c5a6522e19c320cbcfd20af1e70"}
Oct 30 22:23:36.453246 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:36.453221    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-c24d61-5b9ccff96c-klgks"
Oct 30 22:23:36.564916 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:36.564866    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c24d61-5b86d448c-69ql5" podStartSLOduration=1.5648510949999999 podStartE2EDuration="1.564851095s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:23:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:23:36.562436471 +0000 UTC m=+1994.166744025" watchObservedRunningTime="2026-10-30 22:23:36.564851095 +0000 UTC m=+1994.169158695"
Oct 30 22:23:36.702157 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:36.696715    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-c24d61-5b9ccff96c-klgks" podStartSLOduration=1.696682033 podStartE2EDuration="1.696682033s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:23:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:23:36.685748603 +0000 UTC m=+1994.290056159" watchObservedRunningTime="2026-10-30 22:23:36.696682033 +0000 UTC m=+1994.300989587"
Oct 30 22:23:38.302050 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:38.302012    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-99304b-6fb7c9447f-8bncq"]
Oct 30 22:23:38.306756 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:38.306735    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-99304b-6fb7c9447f-8bncq"
Oct 30 22:23:38.361823 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:38.361789    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0a3b40e-76ff-446a-b81c-4278ea7ce23f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-99304b-6fb7c9447f-8bncq\" (UID: \"e0a3b40e-76ff-446a-b81c-4278ea7ce23f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-99304b-6fb7c9447f-8bncq"
Oct 30 22:23:38.361987 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:38.361849    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjt9q\" (UniqueName: \"kubernetes.io/projected/e0a3b40e-76ff-446a-b81c-4278ea7ce23f-kube-api-access-kjt9q\") pod \"maas-api-e2e-shared-b-w5-99304b-6fb7c9447f-8bncq\" (UID: \"e0a3b40e-76ff-446a-b81c-4278ea7ce23f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-99304b-6fb7c9447f-8bncq"
Oct 30 22:23:38.361987 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:38.361926    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0a3b40e-76ff-446a-b81c-4278ea7ce23f-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-99304b-6fb7c9447f-8bncq\" (UID: \"e0a3b40e-76ff-446a-b81c-4278ea7ce23f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-99304b-6fb7c9447f-8bncq"
Oct 30 22:23:38.362162 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:38.362105    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0a3b40e-76ff-446a-b81c-4278ea7ce23f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-99304b-6fb7c9447f-8bncq\" (UID: \"e0a3b40e-76ff-446a-b81c-4278ea7ce23f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-99304b-6fb7c9447f-8bncq"
Oct 30 22:23:38.405343 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:38.405306    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w5-99304b-serving-cert\""
Oct 30 22:23:38.409352 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:38.409324    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-99304b-6fb7c9447f-8bncq"]
Oct 30 22:23:38.463188 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:38.463156    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0a3b40e-76ff-446a-b81c-4278ea7ce23f-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-99304b-6fb7c9447f-8bncq\" (UID: \"e0a3b40e-76ff-446a-b81c-4278ea7ce23f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-99304b-6fb7c9447f-8bncq"
Oct 30 22:23:38.463344 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:38.463281    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0a3b40e-76ff-446a-b81c-4278ea7ce23f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-99304b-6fb7c9447f-8bncq\" (UID: \"e0a3b40e-76ff-446a-b81c-4278ea7ce23f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-99304b-6fb7c9447f-8bncq"
Oct 30 22:23:38.463473 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:38.463457    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0a3b40e-76ff-446a-b81c-4278ea7ce23f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-99304b-6fb7c9447f-8bncq\" (UID: \"e0a3b40e-76ff-446a-b81c-4278ea7ce23f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-99304b-6fb7c9447f-8bncq"
Oct 30 22:23:38.463542 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:38.463526    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjt9q\" (UniqueName: \"kubernetes.io/projected/e0a3b40e-76ff-446a-b81c-4278ea7ce23f-kube-api-access-kjt9q\") pod \"maas-api-e2e-shared-b-w5-99304b-6fb7c9447f-8bncq\" (UID: \"e0a3b40e-76ff-446a-b81c-4278ea7ce23f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-99304b-6fb7c9447f-8bncq"
Oct 30 22:23:38.463627 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:23:38.463602    5678 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-99304b-serving-cert: secret "maas-api-e2e-shared-b-w5-99304b-serving-cert" not found
Oct 30 22:23:38.463748 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:23:38.463694    5678 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0a3b40e-76ff-446a-b81c-4278ea7ce23f-maas-api-tls podName:e0a3b40e-76ff-446a-b81c-4278ea7ce23f nodeName:}" failed. No retries permitted until 2026-10-30 22:23:38.963670472 +0000 UTC m=+1996.567978017 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e0a3b40e-76ff-446a-b81c-4278ea7ce23f-maas-api-tls") pod "maas-api-e2e-shared-b-w5-99304b-6fb7c9447f-8bncq" (UID: "e0a3b40e-76ff-446a-b81c-4278ea7ce23f") : secret "maas-api-e2e-shared-b-w5-99304b-serving-cert" not found
Oct 30 22:23:38.464018 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:38.463988    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0a3b40e-76ff-446a-b81c-4278ea7ce23f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-99304b-6fb7c9447f-8bncq\" (UID: \"e0a3b40e-76ff-446a-b81c-4278ea7ce23f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-99304b-6fb7c9447f-8bncq"
Oct 30 22:23:38.465646 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:38.465630    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0a3b40e-76ff-446a-b81c-4278ea7ce23f-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-99304b-6fb7c9447f-8bncq\" (UID: \"e0a3b40e-76ff-446a-b81c-4278ea7ce23f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-99304b-6fb7c9447f-8bncq"
Oct 30 22:23:38.524756 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:38.524725    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjt9q\" (UniqueName: \"kubernetes.io/projected/e0a3b40e-76ff-446a-b81c-4278ea7ce23f-kube-api-access-kjt9q\") pod \"maas-api-e2e-shared-b-w5-99304b-6fb7c9447f-8bncq\" (UID: \"e0a3b40e-76ff-446a-b81c-4278ea7ce23f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-99304b-6fb7c9447f-8bncq"
Oct 30 22:23:38.972330 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:38.972296    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0a3b40e-76ff-446a-b81c-4278ea7ce23f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-99304b-6fb7c9447f-8bncq\" (UID: \"e0a3b40e-76ff-446a-b81c-4278ea7ce23f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-99304b-6fb7c9447f-8bncq"
Oct 30 22:23:38.974737 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:38.974706    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0a3b40e-76ff-446a-b81c-4278ea7ce23f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-99304b-6fb7c9447f-8bncq\" (UID: \"e0a3b40e-76ff-446a-b81c-4278ea7ce23f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-99304b-6fb7c9447f-8bncq"
Oct 30 22:23:39.221613 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:39.221580    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-99304b-6fb7c9447f-8bncq"
Oct 30 22:23:39.616420 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:39.616371    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-99304b-6fb7c9447f-8bncq"]
Oct 30 22:23:39.622159 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:23:39.622133    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0a3b40e_76ff_446a_b81c_4278ea7ce23f.slice/crio-6e6de717f04f7ae1064b67321ceb085f7be0225cf1767eb62e0339fa7f771d4a WatchSource:0}: Error finding container 6e6de717f04f7ae1064b67321ceb085f7be0225cf1767eb62e0339fa7f771d4a: Status 404 returned error can't find the container with id 6e6de717f04f7ae1064b67321ceb085f7be0225cf1767eb62e0339fa7f771d4a
Oct 30 22:23:40.498045 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:40.498013    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-99304b-6fb7c9447f-8bncq" event={"ID":"e0a3b40e-76ff-446a-b81c-4278ea7ce23f","Type":"ContainerStarted","Data":"6e6de717f04f7ae1064b67321ceb085f7be0225cf1767eb62e0339fa7f771d4a"}
Oct 30 22:23:40.576481 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:23:40.576449    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 22:23:41.507141 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:41.507108    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-99304b-6fb7c9447f-8bncq" event={"ID":"e0a3b40e-76ff-446a-b81c-4278ea7ce23f","Type":"ContainerStarted","Data":"d7ca545f052491a2cd16e5883dd805a3c555f548fad9ccfd0f25e347f1ca00a6"}
Oct 30 22:23:41.507612 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:41.507176    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-99304b-6fb7c9447f-8bncq"
Oct 30 22:23:41.625819 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:41.625758    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-99304b-6fb7c9447f-8bncq" podStartSLOduration=2.679583816 podStartE2EDuration="3.625740757s" totalImagesPullingTime="946.156941ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:23:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:23:41.622231542 +0000 UTC m=+1999.226539095" watchObservedRunningTime="2026-10-30 22:23:41.625740757 +0000 UTC m=+1999.230048312"
Oct 30 22:23:47.475955 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:47.475922    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c24d61-5b86d448c-69ql5"
Oct 30 22:23:47.476489 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:47.476014    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-c24d61-5b9ccff96c-klgks"
Oct 30 22:23:47.520876 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:23:47.520847    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-99304b-6fb7c9447f-8bncq"
Oct 30 22:24:02.799259 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:02.799211    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-943d8f-5bf76c64cb-2rgrl"]
Oct 30 22:24:02.804635 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:02.804610    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-943d8f-5bf76c64cb-2rgrl"
Oct 30 22:24:02.847543 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:02.847508    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9fd32b5-0235-4b94-af2d-7c9bfbe2df09-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-943d8f-5bf76c64cb-2rgrl\" (UID: \"a9fd32b5-0235-4b94-af2d-7c9bfbe2df09\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-943d8f-5bf76c64cb-2rgrl"
Oct 30 22:24:02.847711 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:02.847654    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9fd32b5-0235-4b94-af2d-7c9bfbe2df09-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-943d8f-5bf76c64cb-2rgrl\" (UID: \"a9fd32b5-0235-4b94-af2d-7c9bfbe2df09\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-943d8f-5bf76c64cb-2rgrl"
Oct 30 22:24:02.847789 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:02.847762    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ddks\" (UniqueName: \"kubernetes.io/projected/a9fd32b5-0235-4b94-af2d-7c9bfbe2df09-kube-api-access-2ddks\") pod \"maas-api-e2e-shared-b-w1-943d8f-5bf76c64cb-2rgrl\" (UID: \"a9fd32b5-0235-4b94-af2d-7c9bfbe2df09\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-943d8f-5bf76c64cb-2rgrl"
Oct 30 22:24:02.847870 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:02.847855    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9fd32b5-0235-4b94-af2d-7c9bfbe2df09-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-943d8f-5bf76c64cb-2rgrl\" (UID: \"a9fd32b5-0235-4b94-af2d-7c9bfbe2df09\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-943d8f-5bf76c64cb-2rgrl"
Oct 30 22:24:02.854499 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:02.854477    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w1-943d8f-serving-cert\""
Oct 30 22:24:02.884026 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:02.883989    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-943d8f-5bf76c64cb-2rgrl"]
Oct 30 22:24:02.949176 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:02.949141    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9fd32b5-0235-4b94-af2d-7c9bfbe2df09-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-943d8f-5bf76c64cb-2rgrl\" (UID: \"a9fd32b5-0235-4b94-af2d-7c9bfbe2df09\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-943d8f-5bf76c64cb-2rgrl"
Oct 30 22:24:02.949374 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:02.949264    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9fd32b5-0235-4b94-af2d-7c9bfbe2df09-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-943d8f-5bf76c64cb-2rgrl\" (UID: \"a9fd32b5-0235-4b94-af2d-7c9bfbe2df09\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-943d8f-5bf76c64cb-2rgrl"
Oct 30 22:24:02.949374 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:02.949343    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ddks\" (UniqueName: \"kubernetes.io/projected/a9fd32b5-0235-4b94-af2d-7c9bfbe2df09-kube-api-access-2ddks\") pod \"maas-api-e2e-shared-b-w1-943d8f-5bf76c64cb-2rgrl\" (UID: \"a9fd32b5-0235-4b94-af2d-7c9bfbe2df09\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-943d8f-5bf76c64cb-2rgrl"
Oct 30 22:24:02.949555 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:02.949440    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9fd32b5-0235-4b94-af2d-7c9bfbe2df09-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-943d8f-5bf76c64cb-2rgrl\" (UID: \"a9fd32b5-0235-4b94-af2d-7c9bfbe2df09\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-943d8f-5bf76c64cb-2rgrl"
Oct 30 22:24:02.950351 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:02.950324    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9fd32b5-0235-4b94-af2d-7c9bfbe2df09-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-943d8f-5bf76c64cb-2rgrl\" (UID: \"a9fd32b5-0235-4b94-af2d-7c9bfbe2df09\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-943d8f-5bf76c64cb-2rgrl"
Oct 30 22:24:02.950484 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:24:02.950453    5678 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-943d8f-serving-cert: secret "maas-api-e2e-shared-b-w1-943d8f-serving-cert" not found
Oct 30 22:24:02.950554 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:24:02.950538    5678 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9fd32b5-0235-4b94-af2d-7c9bfbe2df09-maas-api-tls podName:a9fd32b5-0235-4b94-af2d-7c9bfbe2df09 nodeName:}" failed. No retries permitted until 2026-10-30 22:24:03.450493278 +0000 UTC m=+2021.054800826 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a9fd32b5-0235-4b94-af2d-7c9bfbe2df09-maas-api-tls") pod "maas-api-e2e-shared-b-w1-943d8f-5bf76c64cb-2rgrl" (UID: "a9fd32b5-0235-4b94-af2d-7c9bfbe2df09") : secret "maas-api-e2e-shared-b-w1-943d8f-serving-cert" not found
Oct 30 22:24:02.959958 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:02.959925    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9fd32b5-0235-4b94-af2d-7c9bfbe2df09-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-943d8f-5bf76c64cb-2rgrl\" (UID: \"a9fd32b5-0235-4b94-af2d-7c9bfbe2df09\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-943d8f-5bf76c64cb-2rgrl"
Oct 30 22:24:03.010038 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:03.010015    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ddks\" (UniqueName: \"kubernetes.io/projected/a9fd32b5-0235-4b94-af2d-7c9bfbe2df09-kube-api-access-2ddks\") pod \"maas-api-e2e-shared-b-w1-943d8f-5bf76c64cb-2rgrl\" (UID: \"a9fd32b5-0235-4b94-af2d-7c9bfbe2df09\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-943d8f-5bf76c64cb-2rgrl"
Oct 30 22:24:03.458292 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:03.458261    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9fd32b5-0235-4b94-af2d-7c9bfbe2df09-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-943d8f-5bf76c64cb-2rgrl\" (UID: \"a9fd32b5-0235-4b94-af2d-7c9bfbe2df09\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-943d8f-5bf76c64cb-2rgrl"
Oct 30 22:24:03.460769 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:03.460739    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9fd32b5-0235-4b94-af2d-7c9bfbe2df09-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-943d8f-5bf76c64cb-2rgrl\" (UID: \"a9fd32b5-0235-4b94-af2d-7c9bfbe2df09\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-943d8f-5bf76c64cb-2rgrl"
Oct 30 22:24:03.717786 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:03.717706    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-943d8f-5bf76c64cb-2rgrl"
Oct 30 22:24:04.108121 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:04.108040    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-943d8f-5bf76c64cb-2rgrl"]
Oct 30 22:24:04.171063 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:24:04.171037    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9fd32b5_0235_4b94_af2d_7c9bfbe2df09.slice/crio-856ff1fe5fceac15974101d1066f24479d74f7661d9850a2b0a7bcd595a9d229 WatchSource:0}: Error finding container 856ff1fe5fceac15974101d1066f24479d74f7661d9850a2b0a7bcd595a9d229: Status 404 returned error can't find the container with id 856ff1fe5fceac15974101d1066f24479d74f7661d9850a2b0a7bcd595a9d229
Oct 30 22:24:04.685768 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:04.685730    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-943d8f-5bf76c64cb-2rgrl" event={"ID":"a9fd32b5-0235-4b94-af2d-7c9bfbe2df09","Type":"ContainerStarted","Data":"856ff1fe5fceac15974101d1066f24479d74f7661d9850a2b0a7bcd595a9d229"}
Oct 30 22:24:05.695347 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:05.695304    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-943d8f-5bf76c64cb-2rgrl" event={"ID":"a9fd32b5-0235-4b94-af2d-7c9bfbe2df09","Type":"ContainerStarted","Data":"f9ab639922e7443a9778c19f2f562146730975743027b90fce6a4cb46c37ae72"}
Oct 30 22:24:05.695773 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:05.695375    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-943d8f-5bf76c64cb-2rgrl"
Oct 30 22:24:05.816484 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:05.816433    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-943d8f-5bf76c64cb-2rgrl" podStartSLOduration=2.911287546 podStartE2EDuration="3.816415883s" totalImagesPullingTime="905.128337ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:24:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:24:05.813542561 +0000 UTC m=+2023.417850115" watchObservedRunningTime="2026-10-30 22:24:05.816415883 +0000 UTC m=+2023.420723427"
Oct 30 22:24:10.572954 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:24:10.572913    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 22:24:11.708683 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:11.708657    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-943d8f-5bf76c64cb-2rgrl"
Oct 30 22:24:12.707098 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:12.707063    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-5637a7-openshift-default-5b47575485-dxr9g"]
Oct 30 22:24:12.713948 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:12.713924    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5637a7-openshift-default-5b47575485-dxr9g"
Oct 30 22:24:12.764440 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:12.764410    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-5637a7-openshift-default-dockercfg-m5lvg\""
Oct 30 22:24:12.803126 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:12.803096    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c5d0357-a8f0-484c-9584-4b0184fd5284-istio-token\") pod \"e2e-inf-a-5637a7-openshift-default-5b47575485-dxr9g\" (UID: \"9c5d0357-a8f0-484c-9584-4b0184fd5284\") " pod="openshift-ingress/e2e-inf-a-5637a7-openshift-default-5b47575485-dxr9g"
Oct 30 22:24:12.803275 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:12.803147    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c5d0357-a8f0-484c-9584-4b0184fd5284-istio-podinfo\") pod \"e2e-inf-a-5637a7-openshift-default-5b47575485-dxr9g\" (UID: \"9c5d0357-a8f0-484c-9584-4b0184fd5284\") " pod="openshift-ingress/e2e-inf-a-5637a7-openshift-default-5b47575485-dxr9g"
Oct 30 22:24:12.803275 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:12.803256    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c5d0357-a8f0-484c-9584-4b0184fd5284-istio-data\") pod \"e2e-inf-a-5637a7-openshift-default-5b47575485-dxr9g\" (UID: \"9c5d0357-a8f0-484c-9584-4b0184fd5284\") " pod="openshift-ingress/e2e-inf-a-5637a7-openshift-default-5b47575485-dxr9g"
Oct 30 22:24:12.803356 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:12.803310    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c5d0357-a8f0-484c-9584-4b0184fd5284-istio-envoy\") pod \"e2e-inf-a-5637a7-openshift-default-5b47575485-dxr9g\" (UID: \"9c5d0357-a8f0-484c-9584-4b0184fd5284\") " pod="openshift-ingress/e2e-inf-a-5637a7-openshift-default-5b47575485-dxr9g"
Oct 30 22:24:12.803456 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:12.803436    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8kqp\" (UniqueName: \"kubernetes.io/projected/9c5d0357-a8f0-484c-9584-4b0184fd5284-kube-api-access-s8kqp\") pod \"e2e-inf-a-5637a7-openshift-default-5b47575485-dxr9g\" (UID: \"9c5d0357-a8f0-484c-9584-4b0184fd5284\") " pod="openshift-ingress/e2e-inf-a-5637a7-openshift-default-5b47575485-dxr9g"
Oct 30 22:24:12.803572 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:12.803557    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c5d0357-a8f0-484c-9584-4b0184fd5284-credential-socket\") pod \"e2e-inf-a-5637a7-openshift-default-5b47575485-dxr9g\" (UID: \"9c5d0357-a8f0-484c-9584-4b0184fd5284\") " pod="openshift-ingress/e2e-inf-a-5637a7-openshift-default-5b47575485-dxr9g"
Oct 30 22:24:12.803642 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:12.803582    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c5d0357-a8f0-484c-9584-4b0184fd5284-workload-socket\") pod \"e2e-inf-a-5637a7-openshift-default-5b47575485-dxr9g\" (UID: \"9c5d0357-a8f0-484c-9584-4b0184fd5284\") " pod="openshift-ingress/e2e-inf-a-5637a7-openshift-default-5b47575485-dxr9g"
Oct 30 22:24:12.803642 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:12.803604    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c5d0357-a8f0-484c-9584-4b0184fd5284-istiod-ca-cert\") pod \"e2e-inf-a-5637a7-openshift-default-5b47575485-dxr9g\" (UID: \"9c5d0357-a8f0-484c-9584-4b0184fd5284\") " pod="openshift-ingress/e2e-inf-a-5637a7-openshift-default-5b47575485-dxr9g"
Oct 30 22:24:12.803754 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:12.803721    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c5d0357-a8f0-484c-9584-4b0184fd5284-workload-certs\") pod \"e2e-inf-a-5637a7-openshift-default-5b47575485-dxr9g\" (UID: \"9c5d0357-a8f0-484c-9584-4b0184fd5284\") " pod="openshift-ingress/e2e-inf-a-5637a7-openshift-default-5b47575485-dxr9g"
Oct 30 22:24:12.817228 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:12.817201    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-5637a7-openshift-default-5b47575485-dxr9g"]
Oct 30 22:24:12.962480 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:12.949321    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8kqp\" (UniqueName: \"kubernetes.io/projected/9c5d0357-a8f0-484c-9584-4b0184fd5284-kube-api-access-s8kqp\") pod \"e2e-inf-a-5637a7-openshift-default-5b47575485-dxr9g\" (UID: \"9c5d0357-a8f0-484c-9584-4b0184fd5284\") " pod="openshift-ingress/e2e-inf-a-5637a7-openshift-default-5b47575485-dxr9g"
Oct 30 22:24:12.984445 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:12.963250    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c5d0357-a8f0-484c-9584-4b0184fd5284-credential-socket\") pod \"e2e-inf-a-5637a7-openshift-default-5b47575485-dxr9g\" (UID: \"9c5d0357-a8f0-484c-9584-4b0184fd5284\") " pod="openshift-ingress/e2e-inf-a-5637a7-openshift-default-5b47575485-dxr9g"
Oct 30 22:24:12.984769 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:12.984743    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c5d0357-a8f0-484c-9584-4b0184fd5284-workload-socket\") pod \"e2e-inf-a-5637a7-openshift-default-5b47575485-dxr9g\" (UID: \"9c5d0357-a8f0-484c-9584-4b0184fd5284\") " pod="openshift-ingress/e2e-inf-a-5637a7-openshift-default-5b47575485-dxr9g"
Oct 30 22:24:12.999551 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:12.999514    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c5d0357-a8f0-484c-9584-4b0184fd5284-istiod-ca-cert\") pod \"e2e-inf-a-5637a7-openshift-default-5b47575485-dxr9g\" (UID: \"9c5d0357-a8f0-484c-9584-4b0184fd5284\") " pod="openshift-ingress/e2e-inf-a-5637a7-openshift-default-5b47575485-dxr9g"
Oct 30 22:24:13.004553 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:13.004520    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c5d0357-a8f0-484c-9584-4b0184fd5284-workload-certs\") pod \"e2e-inf-a-5637a7-openshift-default-5b47575485-dxr9g\" (UID: \"9c5d0357-a8f0-484c-9584-4b0184fd5284\") " pod="openshift-ingress/e2e-inf-a-5637a7-openshift-default-5b47575485-dxr9g"
Oct 30 22:24:13.013568 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:13.013537    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c5d0357-a8f0-484c-9584-4b0184fd5284-istio-token\") pod \"e2e-inf-a-5637a7-openshift-default-5b47575485-dxr9g\" (UID: \"9c5d0357-a8f0-484c-9584-4b0184fd5284\") " pod="openshift-ingress/e2e-inf-a-5637a7-openshift-default-5b47575485-dxr9g"
Oct 30 22:24:13.071040 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:13.057590    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c5d0357-a8f0-484c-9584-4b0184fd5284-istio-podinfo\") pod \"e2e-inf-a-5637a7-openshift-default-5b47575485-dxr9g\" (UID: \"9c5d0357-a8f0-484c-9584-4b0184fd5284\") " pod="openshift-ingress/e2e-inf-a-5637a7-openshift-default-5b47575485-dxr9g"
Oct 30 22:24:13.071040 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:13.057722    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c5d0357-a8f0-484c-9584-4b0184fd5284-istio-data\") pod \"e2e-inf-a-5637a7-openshift-default-5b47575485-dxr9g\" (UID: \"9c5d0357-a8f0-484c-9584-4b0184fd5284\") " pod="openshift-ingress/e2e-inf-a-5637a7-openshift-default-5b47575485-dxr9g"
Oct 30 22:24:13.071040 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:13.057813    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c5d0357-a8f0-484c-9584-4b0184fd5284-istio-envoy\") pod \"e2e-inf-a-5637a7-openshift-default-5b47575485-dxr9g\" (UID: \"9c5d0357-a8f0-484c-9584-4b0184fd5284\") " pod="openshift-ingress/e2e-inf-a-5637a7-openshift-default-5b47575485-dxr9g"
Oct 30 22:24:13.071040 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:12.985197    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c5d0357-a8f0-484c-9584-4b0184fd5284-workload-socket\") pod \"e2e-inf-a-5637a7-openshift-default-5b47575485-dxr9g\" (UID: \"9c5d0357-a8f0-484c-9584-4b0184fd5284\") " pod="openshift-ingress/e2e-inf-a-5637a7-openshift-default-5b47575485-dxr9g"
Oct 30 22:24:13.071040 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:13.000486    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c5d0357-a8f0-484c-9584-4b0184fd5284-istiod-ca-cert\") pod \"e2e-inf-a-5637a7-openshift-default-5b47575485-dxr9g\" (UID: \"9c5d0357-a8f0-484c-9584-4b0184fd5284\") " pod="openshift-ingress/e2e-inf-a-5637a7-openshift-default-5b47575485-dxr9g"
Oct 30 22:24:13.071040 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:13.005759    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c5d0357-a8f0-484c-9584-4b0184fd5284-workload-certs\") pod \"e2e-inf-a-5637a7-openshift-default-5b47575485-dxr9g\" (UID: \"9c5d0357-a8f0-484c-9584-4b0184fd5284\") " pod="openshift-ingress/e2e-inf-a-5637a7-openshift-default-5b47575485-dxr9g"
Oct 30 22:24:13.071040 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:12.971329    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c5d0357-a8f0-484c-9584-4b0184fd5284-credential-socket\") pod \"e2e-inf-a-5637a7-openshift-default-5b47575485-dxr9g\" (UID: \"9c5d0357-a8f0-484c-9584-4b0184fd5284\") " pod="openshift-ingress/e2e-inf-a-5637a7-openshift-default-5b47575485-dxr9g"
Oct 30 22:24:13.071040 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:13.065159    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c5d0357-a8f0-484c-9584-4b0184fd5284-istio-data\") pod \"e2e-inf-a-5637a7-openshift-default-5b47575485-dxr9g\" (UID: \"9c5d0357-a8f0-484c-9584-4b0184fd5284\") " pod="openshift-ingress/e2e-inf-a-5637a7-openshift-default-5b47575485-dxr9g"
Oct 30 22:24:13.083429 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:13.074087    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8kqp\" (UniqueName: \"kubernetes.io/projected/9c5d0357-a8f0-484c-9584-4b0184fd5284-kube-api-access-s8kqp\") pod \"e2e-inf-a-5637a7-openshift-default-5b47575485-dxr9g\" (UID: \"9c5d0357-a8f0-484c-9584-4b0184fd5284\") " pod="openshift-ingress/e2e-inf-a-5637a7-openshift-default-5b47575485-dxr9g"
Oct 30 22:24:13.083678 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:13.076901    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c5d0357-a8f0-484c-9584-4b0184fd5284-istio-envoy\") pod \"e2e-inf-a-5637a7-openshift-default-5b47575485-dxr9g\" (UID: \"9c5d0357-a8f0-484c-9584-4b0184fd5284\") " pod="openshift-ingress/e2e-inf-a-5637a7-openshift-default-5b47575485-dxr9g"
Oct 30 22:24:13.083799 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:13.078241    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c5d0357-a8f0-484c-9584-4b0184fd5284-istio-token\") pod \"e2e-inf-a-5637a7-openshift-default-5b47575485-dxr9g\" (UID: \"9c5d0357-a8f0-484c-9584-4b0184fd5284\") " pod="openshift-ingress/e2e-inf-a-5637a7-openshift-default-5b47575485-dxr9g"
Oct 30 22:24:13.092420 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:13.084701    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c5d0357-a8f0-484c-9584-4b0184fd5284-istio-podinfo\") pod \"e2e-inf-a-5637a7-openshift-default-5b47575485-dxr9g\" (UID: \"9c5d0357-a8f0-484c-9584-4b0184fd5284\") " pod="openshift-ingress/e2e-inf-a-5637a7-openshift-default-5b47575485-dxr9g"
Oct 30 22:24:13.333647 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:13.333571    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5637a7-openshift-default-5b47575485-dxr9g"
Oct 30 22:24:13.799911 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:13.799882    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-5637a7-openshift-default-5b47575485-dxr9g"]
Oct 30 22:24:13.805076 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:24:13.805044    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c5d0357_a8f0_484c_9584_4b0184fd5284.slice/crio-83c1f197f9a6d0e20ae28b6809e9bdc55d8f9cdd09b6fc0f01cc33a288bce3df WatchSource:0}: Error finding container 83c1f197f9a6d0e20ae28b6809e9bdc55d8f9cdd09b6fc0f01cc33a288bce3df: Status 404 returned error can't find the container with id 83c1f197f9a6d0e20ae28b6809e9bdc55d8f9cdd09b6fc0f01cc33a288bce3df
Oct 30 22:24:13.807635 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:13.807604    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:24:13.807733 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:13.807687    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:24:13.807733 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:13.807716    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:24:14.763476 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:14.763436    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5637a7-openshift-default-5b47575485-dxr9g" event={"ID":"9c5d0357-a8f0-484c-9584-4b0184fd5284","Type":"ContainerStarted","Data":"79480cc4e93133cbe7c6c8aa113e4499d86701e9e32b9d50abb75fa7606c27af"}
Oct 30 22:24:14.763650 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:14.763500    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5637a7-openshift-default-5b47575485-dxr9g" event={"ID":"9c5d0357-a8f0-484c-9584-4b0184fd5284","Type":"ContainerStarted","Data":"83c1f197f9a6d0e20ae28b6809e9bdc55d8f9cdd09b6fc0f01cc33a288bce3df"}
Oct 30 22:24:14.929639 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:14.929592    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-5637a7-openshift-default-5b47575485-dxr9g" podStartSLOduration=2.929578404 podStartE2EDuration="2.929578404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:24:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:24:14.926653342 +0000 UTC m=+2032.530960893" watchObservedRunningTime="2026-10-30 22:24:14.929578404 +0000 UTC m=+2032.533885956"
Oct 30 22:24:15.334246 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:15.334213    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-5637a7-openshift-default-5b47575485-dxr9g"
Oct 30 22:24:15.340906 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:15.340880    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-5637a7-openshift-default-5b47575485-dxr9g"
Oct 30 22:24:15.770895 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:15.770864    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-5637a7-openshift-default-5b47575485-dxr9g"
Oct 30 22:24:15.771750 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:15.771733    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-5637a7-openshift-default-5b47575485-dxr9g"
Oct 30 22:24:18.353530 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:18.353496    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-9f3ea1-openshift-default-5b767dc779-rbqjf"]
Oct 30 22:24:18.358786 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:18.358763    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-9f3ea1-openshift-default-5b767dc779-rbqjf"
Oct 30 22:24:18.410631 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:18.410601    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-9f3ea1-openshift-default-dockercfg-x2g4d\""
Oct 30 22:24:18.451229 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:18.451200    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f9052f2-a2a5-47a5-99b8-138d4e59755c-istio-podinfo\") pod \"e2e-shared-b-w2-9f3ea1-openshift-default-5b767dc779-rbqjf\" (UID: \"6f9052f2-a2a5-47a5-99b8-138d4e59755c\") " pod="openshift-ingress/e2e-shared-b-w2-9f3ea1-openshift-default-5b767dc779-rbqjf"
Oct 30 22:24:18.451456 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:18.451236    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f9052f2-a2a5-47a5-99b8-138d4e59755c-istio-data\") pod \"e2e-shared-b-w2-9f3ea1-openshift-default-5b767dc779-rbqjf\" (UID: \"6f9052f2-a2a5-47a5-99b8-138d4e59755c\") " pod="openshift-ingress/e2e-shared-b-w2-9f3ea1-openshift-default-5b767dc779-rbqjf"
Oct 30 22:24:18.451591 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:18.451562    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f9052f2-a2a5-47a5-99b8-138d4e59755c-istiod-ca-cert\") pod \"e2e-shared-b-w2-9f3ea1-openshift-default-5b767dc779-rbqjf\" (UID: \"6f9052f2-a2a5-47a5-99b8-138d4e59755c\") " pod="openshift-ingress/e2e-shared-b-w2-9f3ea1-openshift-default-5b767dc779-rbqjf"
Oct 30 22:24:18.452574 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:18.452539    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f9052f2-a2a5-47a5-99b8-138d4e59755c-credential-socket\") pod \"e2e-shared-b-w2-9f3ea1-openshift-default-5b767dc779-rbqjf\" (UID: \"6f9052f2-a2a5-47a5-99b8-138d4e59755c\") " pod="openshift-ingress/e2e-shared-b-w2-9f3ea1-openshift-default-5b767dc779-rbqjf"
Oct 30 22:24:18.452725 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:18.452702    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f9052f2-a2a5-47a5-99b8-138d4e59755c-istio-token\") pod \"e2e-shared-b-w2-9f3ea1-openshift-default-5b767dc779-rbqjf\" (UID: \"6f9052f2-a2a5-47a5-99b8-138d4e59755c\") " pod="openshift-ingress/e2e-shared-b-w2-9f3ea1-openshift-default-5b767dc779-rbqjf"
Oct 30 22:24:18.452889 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:18.452867    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f9052f2-a2a5-47a5-99b8-138d4e59755c-workload-certs\") pod \"e2e-shared-b-w2-9f3ea1-openshift-default-5b767dc779-rbqjf\" (UID: \"6f9052f2-a2a5-47a5-99b8-138d4e59755c\") " pod="openshift-ingress/e2e-shared-b-w2-9f3ea1-openshift-default-5b767dc779-rbqjf"
Oct 30 22:24:18.452948 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:18.452917    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfj6l\" (UniqueName: \"kubernetes.io/projected/6f9052f2-a2a5-47a5-99b8-138d4e59755c-kube-api-access-hfj6l\") pod \"e2e-shared-b-w2-9f3ea1-openshift-default-5b767dc779-rbqjf\" (UID: \"6f9052f2-a2a5-47a5-99b8-138d4e59755c\") " pod="openshift-ingress/e2e-shared-b-w2-9f3ea1-openshift-default-5b767dc779-rbqjf"
Oct 30 22:24:18.453040 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:18.453025    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f9052f2-a2a5-47a5-99b8-138d4e59755c-istio-envoy\") pod \"e2e-shared-b-w2-9f3ea1-openshift-default-5b767dc779-rbqjf\" (UID: \"6f9052f2-a2a5-47a5-99b8-138d4e59755c\") " pod="openshift-ingress/e2e-shared-b-w2-9f3ea1-openshift-default-5b767dc779-rbqjf"
Oct 30 22:24:18.453101 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:18.453087    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f9052f2-a2a5-47a5-99b8-138d4e59755c-workload-socket\") pod \"e2e-shared-b-w2-9f3ea1-openshift-default-5b767dc779-rbqjf\" (UID: \"6f9052f2-a2a5-47a5-99b8-138d4e59755c\") " pod="openshift-ingress/e2e-shared-b-w2-9f3ea1-openshift-default-5b767dc779-rbqjf"
Oct 30 22:24:18.462145 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:18.462119    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-9f3ea1-openshift-default-5b767dc779-rbqjf"]
Oct 30 22:24:18.566708 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:18.562853    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f9052f2-a2a5-47a5-99b8-138d4e59755c-workload-certs\") pod \"e2e-shared-b-w2-9f3ea1-openshift-default-5b767dc779-rbqjf\" (UID: \"6f9052f2-a2a5-47a5-99b8-138d4e59755c\") " pod="openshift-ingress/e2e-shared-b-w2-9f3ea1-openshift-default-5b767dc779-rbqjf"
Oct 30 22:24:18.566708 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:18.562986    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfj6l\" (UniqueName: \"kubernetes.io/projected/6f9052f2-a2a5-47a5-99b8-138d4e59755c-kube-api-access-hfj6l\") pod \"e2e-shared-b-w2-9f3ea1-openshift-default-5b767dc779-rbqjf\" (UID: \"6f9052f2-a2a5-47a5-99b8-138d4e59755c\") " pod="openshift-ingress/e2e-shared-b-w2-9f3ea1-openshift-default-5b767dc779-rbqjf"
Oct 30 22:24:18.566708 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:18.563090    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f9052f2-a2a5-47a5-99b8-138d4e59755c-istio-envoy\") pod \"e2e-shared-b-w2-9f3ea1-openshift-default-5b767dc779-rbqjf\" (UID: \"6f9052f2-a2a5-47a5-99b8-138d4e59755c\") " pod="openshift-ingress/e2e-shared-b-w2-9f3ea1-openshift-default-5b767dc779-rbqjf"
Oct 30 22:24:18.566708 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:18.563160    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f9052f2-a2a5-47a5-99b8-138d4e59755c-workload-socket\") pod \"e2e-shared-b-w2-9f3ea1-openshift-default-5b767dc779-rbqjf\" (UID: \"6f9052f2-a2a5-47a5-99b8-138d4e59755c\") " pod="openshift-ingress/e2e-shared-b-w2-9f3ea1-openshift-default-5b767dc779-rbqjf"
Oct 30 22:24:18.566708 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:18.563221    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f9052f2-a2a5-47a5-99b8-138d4e59755c-istio-podinfo\") pod \"e2e-shared-b-w2-9f3ea1-openshift-default-5b767dc779-rbqjf\" (UID: \"6f9052f2-a2a5-47a5-99b8-138d4e59755c\") " pod="openshift-ingress/e2e-shared-b-w2-9f3ea1-openshift-default-5b767dc779-rbqjf"
Oct 30 22:24:18.566708 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:18.563256    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f9052f2-a2a5-47a5-99b8-138d4e59755c-istio-data\") pod \"e2e-shared-b-w2-9f3ea1-openshift-default-5b767dc779-rbqjf\" (UID: \"6f9052f2-a2a5-47a5-99b8-138d4e59755c\") " pod="openshift-ingress/e2e-shared-b-w2-9f3ea1-openshift-default-5b767dc779-rbqjf"
Oct 30 22:24:18.566708 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:18.563290    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f9052f2-a2a5-47a5-99b8-138d4e59755c-istiod-ca-cert\") pod \"e2e-shared-b-w2-9f3ea1-openshift-default-5b767dc779-rbqjf\" (UID: \"6f9052f2-a2a5-47a5-99b8-138d4e59755c\") " pod="openshift-ingress/e2e-shared-b-w2-9f3ea1-openshift-default-5b767dc779-rbqjf"
Oct 30 22:24:18.566708 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:18.565072    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f9052f2-a2a5-47a5-99b8-138d4e59755c-credential-socket\") pod \"e2e-shared-b-w2-9f3ea1-openshift-default-5b767dc779-rbqjf\" (UID: \"6f9052f2-a2a5-47a5-99b8-138d4e59755c\") " pod="openshift-ingress/e2e-shared-b-w2-9f3ea1-openshift-default-5b767dc779-rbqjf"
Oct 30 22:24:18.566708 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:18.565180    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f9052f2-a2a5-47a5-99b8-138d4e59755c-istio-token\") pod \"e2e-shared-b-w2-9f3ea1-openshift-default-5b767dc779-rbqjf\" (UID: \"6f9052f2-a2a5-47a5-99b8-138d4e59755c\") " pod="openshift-ingress/e2e-shared-b-w2-9f3ea1-openshift-default-5b767dc779-rbqjf"
Oct 30 22:24:18.566708 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:18.566331    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f9052f2-a2a5-47a5-99b8-138d4e59755c-workload-certs\") pod \"e2e-shared-b-w2-9f3ea1-openshift-default-5b767dc779-rbqjf\" (UID: \"6f9052f2-a2a5-47a5-99b8-138d4e59755c\") " pod="openshift-ingress/e2e-shared-b-w2-9f3ea1-openshift-default-5b767dc779-rbqjf"
Oct 30 22:24:18.573428 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:18.569695    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f9052f2-a2a5-47a5-99b8-138d4e59755c-istio-envoy\") pod \"e2e-shared-b-w2-9f3ea1-openshift-default-5b767dc779-rbqjf\" (UID: \"6f9052f2-a2a5-47a5-99b8-138d4e59755c\") " pod="openshift-ingress/e2e-shared-b-w2-9f3ea1-openshift-default-5b767dc779-rbqjf"
Oct 30 22:24:18.573428 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:18.570324    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f9052f2-a2a5-47a5-99b8-138d4e59755c-istiod-ca-cert\") pod \"e2e-shared-b-w2-9f3ea1-openshift-default-5b767dc779-rbqjf\" (UID: \"6f9052f2-a2a5-47a5-99b8-138d4e59755c\") " pod="openshift-ingress/e2e-shared-b-w2-9f3ea1-openshift-default-5b767dc779-rbqjf"
Oct 30 22:24:18.573428 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:18.570597    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f9052f2-a2a5-47a5-99b8-138d4e59755c-credential-socket\") pod \"e2e-shared-b-w2-9f3ea1-openshift-default-5b767dc779-rbqjf\" (UID: \"6f9052f2-a2a5-47a5-99b8-138d4e59755c\") " pod="openshift-ingress/e2e-shared-b-w2-9f3ea1-openshift-default-5b767dc779-rbqjf"
Oct 30 22:24:18.573428 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:18.570820    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f9052f2-a2a5-47a5-99b8-138d4e59755c-workload-socket\") pod \"e2e-shared-b-w2-9f3ea1-openshift-default-5b767dc779-rbqjf\" (UID: \"6f9052f2-a2a5-47a5-99b8-138d4e59755c\") " pod="openshift-ingress/e2e-shared-b-w2-9f3ea1-openshift-default-5b767dc779-rbqjf"
Oct 30 22:24:18.573772 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:18.573739    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f9052f2-a2a5-47a5-99b8-138d4e59755c-istio-podinfo\") pod \"e2e-shared-b-w2-9f3ea1-openshift-default-5b767dc779-rbqjf\" (UID: \"6f9052f2-a2a5-47a5-99b8-138d4e59755c\") " pod="openshift-ingress/e2e-shared-b-w2-9f3ea1-openshift-default-5b767dc779-rbqjf"
Oct 30 22:24:18.580047 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:18.580020    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f9052f2-a2a5-47a5-99b8-138d4e59755c-istio-data\") pod \"e2e-shared-b-w2-9f3ea1-openshift-default-5b767dc779-rbqjf\" (UID: \"6f9052f2-a2a5-47a5-99b8-138d4e59755c\") " pod="openshift-ingress/e2e-shared-b-w2-9f3ea1-openshift-default-5b767dc779-rbqjf"
Oct 30 22:24:18.626977 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:18.626945    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f9052f2-a2a5-47a5-99b8-138d4e59755c-istio-token\") pod \"e2e-shared-b-w2-9f3ea1-openshift-default-5b767dc779-rbqjf\" (UID: \"6f9052f2-a2a5-47a5-99b8-138d4e59755c\") " pod="openshift-ingress/e2e-shared-b-w2-9f3ea1-openshift-default-5b767dc779-rbqjf"
Oct 30 22:24:18.629045 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:18.629019    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfj6l\" (UniqueName: \"kubernetes.io/projected/6f9052f2-a2a5-47a5-99b8-138d4e59755c-kube-api-access-hfj6l\") pod \"e2e-shared-b-w2-9f3ea1-openshift-default-5b767dc779-rbqjf\" (UID: \"6f9052f2-a2a5-47a5-99b8-138d4e59755c\") " pod="openshift-ingress/e2e-shared-b-w2-9f3ea1-openshift-default-5b767dc779-rbqjf"
Oct 30 22:24:18.674113 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:18.674089    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-9f3ea1-openshift-default-5b767dc779-rbqjf"
Oct 30 22:24:19.140994 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:19.140958    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-9f3ea1-openshift-default-5b767dc779-rbqjf"]
Oct 30 22:24:19.204030 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:24:19.203988    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f9052f2_a2a5_47a5_99b8_138d4e59755c.slice/crio-a9e906d652b5a51d0823a549228b5ff3f0234feb1feb40e79e3e64bc8d95873b WatchSource:0}: Error finding container a9e906d652b5a51d0823a549228b5ff3f0234feb1feb40e79e3e64bc8d95873b: Status 404 returned error can't find the container with id a9e906d652b5a51d0823a549228b5ff3f0234feb1feb40e79e3e64bc8d95873b
Oct 30 22:24:19.206886 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:19.206848    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:24:19.206992 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:19.206919    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:24:19.206992 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:19.206950    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:24:19.806581 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:19.806547    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-9f3ea1-openshift-default-5b767dc779-rbqjf" event={"ID":"6f9052f2-a2a5-47a5-99b8-138d4e59755c","Type":"ContainerStarted","Data":"2bb759a4dd66467471b7996985692e19c6c378e305e2e6d052dcdf00bd1d7daf"}
Oct 30 22:24:19.807027 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:19.806608    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-9f3ea1-openshift-default-5b767dc779-rbqjf" event={"ID":"6f9052f2-a2a5-47a5-99b8-138d4e59755c","Type":"ContainerStarted","Data":"a9e906d652b5a51d0823a549228b5ff3f0234feb1feb40e79e3e64bc8d95873b"}
Oct 30 22:24:20.675267 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:20.675235    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-9f3ea1-openshift-default-5b767dc779-rbqjf"
Oct 30 22:24:20.682290 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:20.682262    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-9f3ea1-openshift-default-5b767dc779-rbqjf"
Oct 30 22:24:20.813793 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:20.813763    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-9f3ea1-openshift-default-5b767dc779-rbqjf"
Oct 30 22:24:20.814786 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:20.814766    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-9f3ea1-openshift-default-5b767dc779-rbqjf"
Oct 30 22:24:20.902087 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:20.902031    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-9f3ea1-openshift-default-5b767dc779-rbqjf" podStartSLOduration=2.902015121 podStartE2EDuration="2.902015121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:24:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:24:19.931838859 +0000 UTC m=+2037.536146414" watchObservedRunningTime="2026-10-30 22:24:20.902015121 +0000 UTC m=+2038.506322747"
Oct 30 22:24:23.602962 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:23.602913    5678 scope.go:122] "RemoveContainer" containerID="02209031bd309f6b04da1ea80db58be2eca85299cbb30f72627a675a7a7a10de"
Oct 30 22:24:40.306272 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:40.306233    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-5686cb-openshift-default-5f6d8dbdbc-s8jnc"]
Oct 30 22:24:40.312038 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:40.312018    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-5686cb-openshift-default-5f6d8dbdbc-s8jnc"
Oct 30 22:24:40.362333 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:40.362302    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-5686cb-openshift-default-dockercfg-h67cz\""
Oct 30 22:24:40.393225 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:40.393190    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/707b0553-ce35-4de0-ba4a-fffa4adfc6ee-istio-podinfo\") pod \"e2e-api-b-5686cb-openshift-default-5f6d8dbdbc-s8jnc\" (UID: \"707b0553-ce35-4de0-ba4a-fffa4adfc6ee\") " pod="openshift-ingress/e2e-api-b-5686cb-openshift-default-5f6d8dbdbc-s8jnc"
Oct 30 22:24:40.393377 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:40.393231    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/707b0553-ce35-4de0-ba4a-fffa4adfc6ee-credential-socket\") pod \"e2e-api-b-5686cb-openshift-default-5f6d8dbdbc-s8jnc\" (UID: \"707b0553-ce35-4de0-ba4a-fffa4adfc6ee\") " pod="openshift-ingress/e2e-api-b-5686cb-openshift-default-5f6d8dbdbc-s8jnc"
Oct 30 22:24:40.393460 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:40.393391    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/707b0553-ce35-4de0-ba4a-fffa4adfc6ee-istio-data\") pod \"e2e-api-b-5686cb-openshift-default-5f6d8dbdbc-s8jnc\" (UID: \"707b0553-ce35-4de0-ba4a-fffa4adfc6ee\") " pod="openshift-ingress/e2e-api-b-5686cb-openshift-default-5f6d8dbdbc-s8jnc"
Oct 30 22:24:40.393534 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:40.393520    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/707b0553-ce35-4de0-ba4a-fffa4adfc6ee-istiod-ca-cert\") pod \"e2e-api-b-5686cb-openshift-default-5f6d8dbdbc-s8jnc\" (UID: \"707b0553-ce35-4de0-ba4a-fffa4adfc6ee\") " pod="openshift-ingress/e2e-api-b-5686cb-openshift-default-5f6d8dbdbc-s8jnc"
Oct 30 22:24:40.393605 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:40.393592    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2xz5\" (UniqueName: \"kubernetes.io/projected/707b0553-ce35-4de0-ba4a-fffa4adfc6ee-kube-api-access-w2xz5\") pod \"e2e-api-b-5686cb-openshift-default-5f6d8dbdbc-s8jnc\" (UID: \"707b0553-ce35-4de0-ba4a-fffa4adfc6ee\") " pod="openshift-ingress/e2e-api-b-5686cb-openshift-default-5f6d8dbdbc-s8jnc"
Oct 30 22:24:40.393693 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:40.393675    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/707b0553-ce35-4de0-ba4a-fffa4adfc6ee-workload-certs\") pod \"e2e-api-b-5686cb-openshift-default-5f6d8dbdbc-s8jnc\" (UID: \"707b0553-ce35-4de0-ba4a-fffa4adfc6ee\") " pod="openshift-ingress/e2e-api-b-5686cb-openshift-default-5f6d8dbdbc-s8jnc"
Oct 30 22:24:40.393728 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:40.393720    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/707b0553-ce35-4de0-ba4a-fffa4adfc6ee-istio-token\") pod \"e2e-api-b-5686cb-openshift-default-5f6d8dbdbc-s8jnc\" (UID: \"707b0553-ce35-4de0-ba4a-fffa4adfc6ee\") " pod="openshift-ingress/e2e-api-b-5686cb-openshift-default-5f6d8dbdbc-s8jnc"
Oct 30 22:24:40.393783 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:40.393769    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/707b0553-ce35-4de0-ba4a-fffa4adfc6ee-workload-socket\") pod \"e2e-api-b-5686cb-openshift-default-5f6d8dbdbc-s8jnc\" (UID: \"707b0553-ce35-4de0-ba4a-fffa4adfc6ee\") " pod="openshift-ingress/e2e-api-b-5686cb-openshift-default-5f6d8dbdbc-s8jnc"
Oct 30 22:24:40.393835 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:40.393792    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/707b0553-ce35-4de0-ba4a-fffa4adfc6ee-istio-envoy\") pod \"e2e-api-b-5686cb-openshift-default-5f6d8dbdbc-s8jnc\" (UID: \"707b0553-ce35-4de0-ba4a-fffa4adfc6ee\") " pod="openshift-ingress/e2e-api-b-5686cb-openshift-default-5f6d8dbdbc-s8jnc"
Oct 30 22:24:40.414581 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:40.414550    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-5686cb-openshift-default-5f6d8dbdbc-s8jnc"]
Oct 30 22:24:40.497613 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:40.497572    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/707b0553-ce35-4de0-ba4a-fffa4adfc6ee-workload-socket\") pod \"e2e-api-b-5686cb-openshift-default-5f6d8dbdbc-s8jnc\" (UID: \"707b0553-ce35-4de0-ba4a-fffa4adfc6ee\") " pod="openshift-ingress/e2e-api-b-5686cb-openshift-default-5f6d8dbdbc-s8jnc"
Oct 30 22:24:40.503597 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:40.503565    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/707b0553-ce35-4de0-ba4a-fffa4adfc6ee-istio-envoy\") pod \"e2e-api-b-5686cb-openshift-default-5f6d8dbdbc-s8jnc\" (UID: \"707b0553-ce35-4de0-ba4a-fffa4adfc6ee\") " pod="openshift-ingress/e2e-api-b-5686cb-openshift-default-5f6d8dbdbc-s8jnc"
Oct 30 22:24:40.504302 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:40.504266    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/707b0553-ce35-4de0-ba4a-fffa4adfc6ee-istio-podinfo\") pod \"e2e-api-b-5686cb-openshift-default-5f6d8dbdbc-s8jnc\" (UID: \"707b0553-ce35-4de0-ba4a-fffa4adfc6ee\") " pod="openshift-ingress/e2e-api-b-5686cb-openshift-default-5f6d8dbdbc-s8jnc"
Oct 30 22:24:40.504953 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:40.504928    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/707b0553-ce35-4de0-ba4a-fffa4adfc6ee-credential-socket\") pod \"e2e-api-b-5686cb-openshift-default-5f6d8dbdbc-s8jnc\" (UID: \"707b0553-ce35-4de0-ba4a-fffa4adfc6ee\") " pod="openshift-ingress/e2e-api-b-5686cb-openshift-default-5f6d8dbdbc-s8jnc"
Oct 30 22:24:40.505092 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:40.498187    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/707b0553-ce35-4de0-ba4a-fffa4adfc6ee-workload-socket\") pod \"e2e-api-b-5686cb-openshift-default-5f6d8dbdbc-s8jnc\" (UID: \"707b0553-ce35-4de0-ba4a-fffa4adfc6ee\") " pod="openshift-ingress/e2e-api-b-5686cb-openshift-default-5f6d8dbdbc-s8jnc"
Oct 30 22:24:40.505231 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:40.505082    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/707b0553-ce35-4de0-ba4a-fffa4adfc6ee-istio-data\") pod \"e2e-api-b-5686cb-openshift-default-5f6d8dbdbc-s8jnc\" (UID: \"707b0553-ce35-4de0-ba4a-fffa4adfc6ee\") " pod="openshift-ingress/e2e-api-b-5686cb-openshift-default-5f6d8dbdbc-s8jnc"
Oct 30 22:24:40.506429 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:40.505325    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/707b0553-ce35-4de0-ba4a-fffa4adfc6ee-istio-data\") pod \"e2e-api-b-5686cb-openshift-default-5f6d8dbdbc-s8jnc\" (UID: \"707b0553-ce35-4de0-ba4a-fffa4adfc6ee\") " pod="openshift-ingress/e2e-api-b-5686cb-openshift-default-5f6d8dbdbc-s8jnc"
Oct 30 22:24:40.506645 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:40.505476    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/707b0553-ce35-4de0-ba4a-fffa4adfc6ee-istiod-ca-cert\") pod \"e2e-api-b-5686cb-openshift-default-5f6d8dbdbc-s8jnc\" (UID: \"707b0553-ce35-4de0-ba4a-fffa4adfc6ee\") " pod="openshift-ingress/e2e-api-b-5686cb-openshift-default-5f6d8dbdbc-s8jnc"
Oct 30 22:24:40.506790 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:40.506725    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2xz5\" (UniqueName: \"kubernetes.io/projected/707b0553-ce35-4de0-ba4a-fffa4adfc6ee-kube-api-access-w2xz5\") pod \"e2e-api-b-5686cb-openshift-default-5f6d8dbdbc-s8jnc\" (UID: \"707b0553-ce35-4de0-ba4a-fffa4adfc6ee\") " pod="openshift-ingress/e2e-api-b-5686cb-openshift-default-5f6d8dbdbc-s8jnc"
Oct 30 22:24:40.506855 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:40.506805    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/707b0553-ce35-4de0-ba4a-fffa4adfc6ee-workload-certs\") pod \"e2e-api-b-5686cb-openshift-default-5f6d8dbdbc-s8jnc\" (UID: \"707b0553-ce35-4de0-ba4a-fffa4adfc6ee\") " pod="openshift-ingress/e2e-api-b-5686cb-openshift-default-5f6d8dbdbc-s8jnc"
Oct 30 22:24:40.506855 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:40.506838    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/707b0553-ce35-4de0-ba4a-fffa4adfc6ee-istio-token\") pod \"e2e-api-b-5686cb-openshift-default-5f6d8dbdbc-s8jnc\" (UID: \"707b0553-ce35-4de0-ba4a-fffa4adfc6ee\") " pod="openshift-ingress/e2e-api-b-5686cb-openshift-default-5f6d8dbdbc-s8jnc"
Oct 30 22:24:40.507212 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:40.506154    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/707b0553-ce35-4de0-ba4a-fffa4adfc6ee-istiod-ca-cert\") pod \"e2e-api-b-5686cb-openshift-default-5f6d8dbdbc-s8jnc\" (UID: \"707b0553-ce35-4de0-ba4a-fffa4adfc6ee\") " pod="openshift-ingress/e2e-api-b-5686cb-openshift-default-5f6d8dbdbc-s8jnc"
Oct 30 22:24:40.507393 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:40.506393    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/707b0553-ce35-4de0-ba4a-fffa4adfc6ee-credential-socket\") pod \"e2e-api-b-5686cb-openshift-default-5f6d8dbdbc-s8jnc\" (UID: \"707b0553-ce35-4de0-ba4a-fffa4adfc6ee\") " pod="openshift-ingress/e2e-api-b-5686cb-openshift-default-5f6d8dbdbc-s8jnc"
Oct 30 22:24:40.507741 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:40.507717    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/707b0553-ce35-4de0-ba4a-fffa4adfc6ee-workload-certs\") pod \"e2e-api-b-5686cb-openshift-default-5f6d8dbdbc-s8jnc\" (UID: \"707b0553-ce35-4de0-ba4a-fffa4adfc6ee\") " pod="openshift-ingress/e2e-api-b-5686cb-openshift-default-5f6d8dbdbc-s8jnc"
Oct 30 22:24:40.522474 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:40.522441    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/707b0553-ce35-4de0-ba4a-fffa4adfc6ee-istio-podinfo\") pod \"e2e-api-b-5686cb-openshift-default-5f6d8dbdbc-s8jnc\" (UID: \"707b0553-ce35-4de0-ba4a-fffa4adfc6ee\") " pod="openshift-ingress/e2e-api-b-5686cb-openshift-default-5f6d8dbdbc-s8jnc"
Oct 30 22:24:40.532073 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:40.531976    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/707b0553-ce35-4de0-ba4a-fffa4adfc6ee-istio-envoy\") pod \"e2e-api-b-5686cb-openshift-default-5f6d8dbdbc-s8jnc\" (UID: \"707b0553-ce35-4de0-ba4a-fffa4adfc6ee\") " pod="openshift-ingress/e2e-api-b-5686cb-openshift-default-5f6d8dbdbc-s8jnc"
Oct 30 22:24:40.567461 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:40.567381    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2xz5\" (UniqueName: \"kubernetes.io/projected/707b0553-ce35-4de0-ba4a-fffa4adfc6ee-kube-api-access-w2xz5\") pod \"e2e-api-b-5686cb-openshift-default-5f6d8dbdbc-s8jnc\" (UID: \"707b0553-ce35-4de0-ba4a-fffa4adfc6ee\") " pod="openshift-ingress/e2e-api-b-5686cb-openshift-default-5f6d8dbdbc-s8jnc"
Oct 30 22:24:40.569136 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:40.569111    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/707b0553-ce35-4de0-ba4a-fffa4adfc6ee-istio-token\") pod \"e2e-api-b-5686cb-openshift-default-5f6d8dbdbc-s8jnc\" (UID: \"707b0553-ce35-4de0-ba4a-fffa4adfc6ee\") " pod="openshift-ingress/e2e-api-b-5686cb-openshift-default-5f6d8dbdbc-s8jnc"
Oct 30 22:24:40.581896 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:24:40.581861    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 22:24:40.626667 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:40.626643    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-5686cb-openshift-default-5f6d8dbdbc-s8jnc"
Oct 30 22:24:41.080722 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:41.080685    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-5686cb-openshift-default-5f6d8dbdbc-s8jnc"]
Oct 30 22:24:41.094074 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:24:41.094044    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod707b0553_ce35_4de0_ba4a_fffa4adfc6ee.slice/crio-4288eaafb1597b99650ed433f410afa0fc814d0e3a04f987be37915cb1a1abde WatchSource:0}: Error finding container 4288eaafb1597b99650ed433f410afa0fc814d0e3a04f987be37915cb1a1abde: Status 404 returned error can't find the container with id 4288eaafb1597b99650ed433f410afa0fc814d0e3a04f987be37915cb1a1abde
Oct 30 22:24:41.097309 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:41.097277    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:24:41.097418 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:41.097342    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:24:41.097418 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:41.097371    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:24:42.068562 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:42.068532    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-5686cb-openshift-default-5f6d8dbdbc-s8jnc" event={"ID":"707b0553-ce35-4de0-ba4a-fffa4adfc6ee","Type":"ContainerStarted","Data":"1dc97660aff8ac9223122df34c70ae1232b4e4ba5e5378e1b72098e2e8e770ec"}
Oct 30 22:24:42.069046 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:42.068585    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-5686cb-openshift-default-5f6d8dbdbc-s8jnc" event={"ID":"707b0553-ce35-4de0-ba4a-fffa4adfc6ee","Type":"ContainerStarted","Data":"4288eaafb1597b99650ed433f410afa0fc814d0e3a04f987be37915cb1a1abde"}
Oct 30 22:24:42.239726 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:42.239655    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-5686cb-openshift-default-5f6d8dbdbc-s8jnc" podStartSLOduration=2.239636465 podStartE2EDuration="2.239636465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:24:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:24:42.236500056 +0000 UTC m=+2059.840807610" watchObservedRunningTime="2026-10-30 22:24:42.239636465 +0000 UTC m=+2059.843944019"
Oct 30 22:24:42.627148 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:42.627117    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-5686cb-openshift-default-5f6d8dbdbc-s8jnc"
Oct 30 22:24:42.632579 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:42.632552    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-5686cb-openshift-default-5f6d8dbdbc-s8jnc"
Oct 30 22:24:43.080106 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:43.080029    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-5686cb-openshift-default-5f6d8dbdbc-s8jnc"
Oct 30 22:24:43.081009 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:24:43.080987    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-5686cb-openshift-default-5f6d8dbdbc-s8jnc"
Oct 30 22:25:08.993662 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:25:08.993586    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-l8b5h"]
Oct 30 22:25:08.994175 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:25:08.993788    5678 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-l8b5h" podUID="09fbaa51-372b-406e-a478-3a7d52a8a618" containerName="istio-proxy" containerID="cri-o://79803bb8b68800167f26faf4dc73405ced63ea2b0300cc93776be9cec8f69d84" gracePeriod=30
Oct 30 22:25:10.203138 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:25:10.203100    5678 patch_prober.go:29] interesting pod/e2e-worker-w2-bc714d-openshift-default-67558b95cf-l8b5h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:25:10.203620 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:25:10.203160    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-l8b5h" podUID="09fbaa51-372b-406e-a478-3a7d52a8a618" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:25:10.572786 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:25:10.572706    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 22:25:14.276914 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:25:14.276849    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-l8b5h"
Oct 30 22:25:14.343723 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:25:14.343689    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09fbaa51-372b-406e-a478-3a7d52a8a618-kube-api-access-fv2sh\" (UniqueName: \"kubernetes.io/projected/09fbaa51-372b-406e-a478-3a7d52a8a618-kube-api-access-fv2sh\") pod \"09fbaa51-372b-406e-a478-3a7d52a8a618\" (UID: \"09fbaa51-372b-406e-a478-3a7d52a8a618\") "
Oct 30 22:25:14.343887 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:25:14.343760    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09fbaa51-372b-406e-a478-3a7d52a8a618-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09fbaa51-372b-406e-a478-3a7d52a8a618-credential-socket\") pod \"09fbaa51-372b-406e-a478-3a7d52a8a618\" (UID: \"09fbaa51-372b-406e-a478-3a7d52a8a618\") "
Oct 30 22:25:14.343887 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:25:14.343825    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09fbaa51-372b-406e-a478-3a7d52a8a618-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09fbaa51-372b-406e-a478-3a7d52a8a618-istio-data\") pod \"09fbaa51-372b-406e-a478-3a7d52a8a618\" (UID: \"09fbaa51-372b-406e-a478-3a7d52a8a618\") "
Oct 30 22:25:14.343887 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:25:14.343851    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09fbaa51-372b-406e-a478-3a7d52a8a618-istio-token\" (UniqueName: \"kubernetes.io/projected/09fbaa51-372b-406e-a478-3a7d52a8a618-istio-token\") pod \"09fbaa51-372b-406e-a478-3a7d52a8a618\" (UID: \"09fbaa51-372b-406e-a478-3a7d52a8a618\") "
Oct 30 22:25:14.344079 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:25:14.343946    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09fbaa51-372b-406e-a478-3a7d52a8a618-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09fbaa51-372b-406e-a478-3a7d52a8a618-workload-socket\") pod \"09fbaa51-372b-406e-a478-3a7d52a8a618\" (UID: \"09fbaa51-372b-406e-a478-3a7d52a8a618\") "
Oct 30 22:25:14.344079 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:25:14.344017    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/09fbaa51-372b-406e-a478-3a7d52a8a618-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09fbaa51-372b-406e-a478-3a7d52a8a618-istio-podinfo\") pod \"09fbaa51-372b-406e-a478-3a7d52a8a618\" (UID: \"09fbaa51-372b-406e-a478-3a7d52a8a618\") "
Oct 30 22:25:14.344079 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:25:14.344045    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09fbaa51-372b-406e-a478-3a7d52a8a618-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09fbaa51-372b-406e-a478-3a7d52a8a618-istio-envoy\") pod \"09fbaa51-372b-406e-a478-3a7d52a8a618\" (UID: \"09fbaa51-372b-406e-a478-3a7d52a8a618\") "
Oct 30 22:25:14.344219 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:25:14.344111    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09fbaa51-372b-406e-a478-3a7d52a8a618-credential-socket" pod "09fbaa51-372b-406e-a478-3a7d52a8a618" (UID: "09fbaa51-372b-406e-a478-3a7d52a8a618"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:25:14.344219 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:25:14.344154    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09fbaa51-372b-406e-a478-3a7d52a8a618-istio-data" pod "09fbaa51-372b-406e-a478-3a7d52a8a618" (UID: "09fbaa51-372b-406e-a478-3a7d52a8a618"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:25:14.344219 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:25:14.344192    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09fbaa51-372b-406e-a478-3a7d52a8a618-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09fbaa51-372b-406e-a478-3a7d52a8a618-istiod-ca-cert\") pod \"09fbaa51-372b-406e-a478-3a7d52a8a618\" (UID: \"09fbaa51-372b-406e-a478-3a7d52a8a618\") "
Oct 30 22:25:14.344219 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:25:14.344212    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09fbaa51-372b-406e-a478-3a7d52a8a618-workload-socket" pod "09fbaa51-372b-406e-a478-3a7d52a8a618" (UID: "09fbaa51-372b-406e-a478-3a7d52a8a618"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:25:14.344454 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:25:14.344345    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09fbaa51-372b-406e-a478-3a7d52a8a618-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09fbaa51-372b-406e-a478-3a7d52a8a618-workload-certs\") pod \"09fbaa51-372b-406e-a478-3a7d52a8a618\" (UID: \"09fbaa51-372b-406e-a478-3a7d52a8a618\") "
Oct 30 22:25:14.344666 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:25:14.344590    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09fbaa51-372b-406e-a478-3a7d52a8a618-istiod-ca-cert" pod "09fbaa51-372b-406e-a478-3a7d52a8a618" (UID: "09fbaa51-372b-406e-a478-3a7d52a8a618"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:25:14.344767 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:25:14.344668    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09fbaa51-372b-406e-a478-3a7d52a8a618-workload-certs" pod "09fbaa51-372b-406e-a478-3a7d52a8a618" (UID: "09fbaa51-372b-406e-a478-3a7d52a8a618"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:25:14.345208 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:25:14.345185    5678 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09fbaa51-372b-406e-a478-3a7d52a8a618-workload-certs\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:25:14.345257 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:25:14.345219    5678 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09fbaa51-372b-406e-a478-3a7d52a8a618-credential-socket\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:25:14.345257 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:25:14.345234    5678 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09fbaa51-372b-406e-a478-3a7d52a8a618-istio-data\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:25:14.345257 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:25:14.345248    5678 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09fbaa51-372b-406e-a478-3a7d52a8a618-workload-socket\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:25:14.345363 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:25:14.345261    5678 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09fbaa51-372b-406e-a478-3a7d52a8a618-istiod-ca-cert\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:25:14.346289 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:25:14.346263    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09fbaa51-372b-406e-a478-3a7d52a8a618-kube-api-access-fv2sh" pod "09fbaa51-372b-406e-a478-3a7d52a8a618" (UID: "09fbaa51-372b-406e-a478-3a7d52a8a618"). InnerVolumeSpecName "kube-api-access-fv2sh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:25:14.346424 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:25:14.346301    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/09fbaa51-372b-406e-a478-3a7d52a8a618-istio-podinfo" pod "09fbaa51-372b-406e-a478-3a7d52a8a618" (UID: "09fbaa51-372b-406e-a478-3a7d52a8a618"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:25:14.346624 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:25:14.346601    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09fbaa51-372b-406e-a478-3a7d52a8a618-istio-token" pod "09fbaa51-372b-406e-a478-3a7d52a8a618" (UID: "09fbaa51-372b-406e-a478-3a7d52a8a618"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:25:14.346700 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:25:14.346601    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09fbaa51-372b-406e-a478-3a7d52a8a618-istio-envoy" pod "09fbaa51-372b-406e-a478-3a7d52a8a618" (UID: "09fbaa51-372b-406e-a478-3a7d52a8a618"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:25:14.374980 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:25:14.374954    5678 generic.go:356] "Generic (PLEG): container finished" podID="09fbaa51-372b-406e-a478-3a7d52a8a618" containerID="79803bb8b68800167f26faf4dc73405ced63ea2b0300cc93776be9cec8f69d84" exitCode=0
Oct 30 22:25:14.375149 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:25:14.375001    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-l8b5h"
Oct 30 22:25:14.375149 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:25:14.375038    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-l8b5h" event={"ID":"09fbaa51-372b-406e-a478-3a7d52a8a618","Type":"ContainerDied","Data":"79803bb8b68800167f26faf4dc73405ced63ea2b0300cc93776be9cec8f69d84"}
Oct 30 22:25:14.375149 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:25:14.375125    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-l8b5h" event={"ID":"09fbaa51-372b-406e-a478-3a7d52a8a618","Type":"ContainerDied","Data":"1e723b77f17c825c0303143b2634e378c6dba82165ab2b3a38a3d5a7398a5ab4"}
Oct 30 22:25:14.375268 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:25:14.375153    5678 scope.go:122] "RemoveContainer" containerID="79803bb8b68800167f26faf4dc73405ced63ea2b0300cc93776be9cec8f69d84"
Oct 30 22:25:14.386004 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:25:14.385987    5678 scope.go:122] "RemoveContainer" containerID="79803bb8b68800167f26faf4dc73405ced63ea2b0300cc93776be9cec8f69d84"
Oct 30 22:25:14.386281 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:25:14.386263    5678 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79803bb8b68800167f26faf4dc73405ced63ea2b0300cc93776be9cec8f69d84\": container with ID starting with 79803bb8b68800167f26faf4dc73405ced63ea2b0300cc93776be9cec8f69d84 not found: ID does not exist" containerID="79803bb8b68800167f26faf4dc73405ced63ea2b0300cc93776be9cec8f69d84"
Oct 30 22:25:14.386349 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:25:14.386295    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79803bb8b68800167f26faf4dc73405ced63ea2b0300cc93776be9cec8f69d84"} err="failed to get container status \"79803bb8b68800167f26faf4dc73405ced63ea2b0300cc93776be9cec8f69d84\": rpc error: code = NotFound desc = could not find container \"79803bb8b68800167f26faf4dc73405ced63ea2b0300cc93776be9cec8f69d84\": container with ID starting with 79803bb8b68800167f26faf4dc73405ced63ea2b0300cc93776be9cec8f69d84 not found: ID does not exist"
Oct 30 22:25:14.447186 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:25:14.447147    5678 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09fbaa51-372b-406e-a478-3a7d52a8a618-istio-token\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:25:14.447186 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:25:14.447186    5678 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09fbaa51-372b-406e-a478-3a7d52a8a618-istio-podinfo\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:25:14.447354 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:25:14.447202    5678 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09fbaa51-372b-406e-a478-3a7d52a8a618-istio-envoy\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:25:14.447354 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:25:14.447214    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fv2sh\" (UniqueName: \"kubernetes.io/projected/09fbaa51-372b-406e-a478-3a7d52a8a618-kube-api-access-fv2sh\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:25:14.661552 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:25:14.661521    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-l8b5h"]
Oct 30 22:25:14.667730 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:25:14.667701    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-l8b5h"]
Oct 30 22:25:16.526011 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:25:16.525979    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09fbaa51-372b-406e-a478-3a7d52a8a618" path="/var/lib/kubelet/pods/09fbaa51-372b-406e-a478-3a7d52a8a618/volumes"
Oct 30 22:25:40.573648 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:25:40.573604    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 22:26:03.061861 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:03.061828    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a98f5ef3-rtq8s"]
Oct 30 22:26:03.109096 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:03.109068    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a98f5ef3-rtq8s"]
Oct 30 22:26:04.524940 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:04.524904    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="defca364-7e29-483a-b948-d41e8bb31cd9" path="/var/lib/kubelet/pods/defca364-7e29-483a-b948-d41e8bb31cd9/volumes"
Oct 30 22:26:09.481315 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:09.481281    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9bfdx"]
Oct 30 22:26:09.481968 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:09.481530    5678 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9bfdx" podUID="1b614bbd-09b5-4b79-a9a5-1c57ae795852" containerName="discovery" containerID="cri-o://8afe92c9b204030cfad6f3dc612d17be23f6d48382a88b78b930024f42bd1235" gracePeriod=30
Oct 30 22:26:09.779945 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:09.779916    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9bfdx"
Oct 30 22:26:09.837768 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:09.837738    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b614bbd-09b5-4b79-a9a5-1c57ae795852-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1b614bbd-09b5-4b79-a9a5-1c57ae795852-istio-kubeconfig\") pod \"1b614bbd-09b5-4b79-a9a5-1c57ae795852\" (UID: \"1b614bbd-09b5-4b79-a9a5-1c57ae795852\") "
Oct 30 22:26:09.837768 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:09.837771    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b614bbd-09b5-4b79-a9a5-1c57ae795852-kube-api-access-zn9pc\" (UniqueName: \"kubernetes.io/projected/1b614bbd-09b5-4b79-a9a5-1c57ae795852-kube-api-access-zn9pc\") pod \"1b614bbd-09b5-4b79-a9a5-1c57ae795852\" (UID: \"1b614bbd-09b5-4b79-a9a5-1c57ae795852\") "
Oct 30 22:26:09.837967 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:09.837794    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b614bbd-09b5-4b79-a9a5-1c57ae795852-istio-token\" (UniqueName: \"kubernetes.io/projected/1b614bbd-09b5-4b79-a9a5-1c57ae795852-istio-token\") pod \"1b614bbd-09b5-4b79-a9a5-1c57ae795852\" (UID: \"1b614bbd-09b5-4b79-a9a5-1c57ae795852\") "
Oct 30 22:26:09.837967 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:09.837882    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b614bbd-09b5-4b79-a9a5-1c57ae795852-cacerts\" (UniqueName: \"kubernetes.io/secret/1b614bbd-09b5-4b79-a9a5-1c57ae795852-cacerts\") pod \"1b614bbd-09b5-4b79-a9a5-1c57ae795852\" (UID: \"1b614bbd-09b5-4b79-a9a5-1c57ae795852\") "
Oct 30 22:26:09.837967 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:09.837919    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b614bbd-09b5-4b79-a9a5-1c57ae795852-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b614bbd-09b5-4b79-a9a5-1c57ae795852-local-certs\") pod \"1b614bbd-09b5-4b79-a9a5-1c57ae795852\" (UID: \"1b614bbd-09b5-4b79-a9a5-1c57ae795852\") "
Oct 30 22:26:09.838140 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:09.838067    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b614bbd-09b5-4b79-a9a5-1c57ae795852-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1b614bbd-09b5-4b79-a9a5-1c57ae795852-istio-csr-dns-cert\") pod \"1b614bbd-09b5-4b79-a9a5-1c57ae795852\" (UID: \"1b614bbd-09b5-4b79-a9a5-1c57ae795852\") "
Oct 30 22:26:09.838140 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:09.838103    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b614bbd-09b5-4b79-a9a5-1c57ae795852-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1b614bbd-09b5-4b79-a9a5-1c57ae795852-istio-csr-ca-configmap\") pod \"1b614bbd-09b5-4b79-a9a5-1c57ae795852\" (UID: \"1b614bbd-09b5-4b79-a9a5-1c57ae795852\") "
Oct 30 22:26:09.838747 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:09.838708    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b614bbd-09b5-4b79-a9a5-1c57ae795852-istio-csr-ca-configmap" pod "1b614bbd-09b5-4b79-a9a5-1c57ae795852" (UID: "1b614bbd-09b5-4b79-a9a5-1c57ae795852"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:26:09.838956 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:09.838933    5678 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1b614bbd-09b5-4b79-a9a5-1c57ae795852-istio-csr-ca-configmap\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:26:09.840338 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:09.840311    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b614bbd-09b5-4b79-a9a5-1c57ae795852-istio-token" pod "1b614bbd-09b5-4b79-a9a5-1c57ae795852" (UID: "1b614bbd-09b5-4b79-a9a5-1c57ae795852"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:26:09.840458 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:09.840334    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b614bbd-09b5-4b79-a9a5-1c57ae795852-cacerts" pod "1b614bbd-09b5-4b79-a9a5-1c57ae795852" (UID: "1b614bbd-09b5-4b79-a9a5-1c57ae795852"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:26:09.840651 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:09.840624    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b614bbd-09b5-4b79-a9a5-1c57ae795852-istio-csr-dns-cert" pod "1b614bbd-09b5-4b79-a9a5-1c57ae795852" (UID: "1b614bbd-09b5-4b79-a9a5-1c57ae795852"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:26:09.840755 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:09.840647    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b614bbd-09b5-4b79-a9a5-1c57ae795852-local-certs" pod "1b614bbd-09b5-4b79-a9a5-1c57ae795852" (UID: "1b614bbd-09b5-4b79-a9a5-1c57ae795852"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:26:09.840755 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:09.840670    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b614bbd-09b5-4b79-a9a5-1c57ae795852-kube-api-access-zn9pc" pod "1b614bbd-09b5-4b79-a9a5-1c57ae795852" (UID: "1b614bbd-09b5-4b79-a9a5-1c57ae795852"). InnerVolumeSpecName "kube-api-access-zn9pc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:26:09.840755 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:09.840747    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b614bbd-09b5-4b79-a9a5-1c57ae795852-istio-kubeconfig" pod "1b614bbd-09b5-4b79-a9a5-1c57ae795852" (UID: "1b614bbd-09b5-4b79-a9a5-1c57ae795852"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:26:09.940459 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:09.940318    5678 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1b614bbd-09b5-4b79-a9a5-1c57ae795852-istio-kubeconfig\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:26:09.940459 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:09.940357    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zn9pc\" (UniqueName: \"kubernetes.io/projected/1b614bbd-09b5-4b79-a9a5-1c57ae795852-kube-api-access-zn9pc\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:26:09.940459 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:09.940375    5678 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b614bbd-09b5-4b79-a9a5-1c57ae795852-istio-token\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:26:09.940459 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:09.940389    5678 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1b614bbd-09b5-4b79-a9a5-1c57ae795852-cacerts\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:26:09.940459 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:09.940419    5678 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b614bbd-09b5-4b79-a9a5-1c57ae795852-local-certs\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:26:09.940459 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:09.940433    5678 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1b614bbd-09b5-4b79-a9a5-1c57ae795852-istio-csr-dns-cert\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:26:09.997176 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:09.997090    5678 generic.go:356] "Generic (PLEG): container finished" podID="1b614bbd-09b5-4b79-a9a5-1c57ae795852" containerID="8afe92c9b204030cfad6f3dc612d17be23f6d48382a88b78b930024f42bd1235" exitCode=0
Oct 30 22:26:09.997176 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:09.997137    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9bfdx"
Oct 30 22:26:09.997359 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:09.997175    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9bfdx" event={"ID":"1b614bbd-09b5-4b79-a9a5-1c57ae795852","Type":"ContainerDied","Data":"8afe92c9b204030cfad6f3dc612d17be23f6d48382a88b78b930024f42bd1235"}
Oct 30 22:26:09.997359 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:09.997227    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9bfdx" event={"ID":"1b614bbd-09b5-4b79-a9a5-1c57ae795852","Type":"ContainerDied","Data":"33234d2431251f16d37364e70b09ff26814413778ecd249b29aebb8bcfd89afa"}
Oct 30 22:26:09.997359 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:09.997249    5678 scope.go:122] "RemoveContainer" containerID="8afe92c9b204030cfad6f3dc612d17be23f6d48382a88b78b930024f42bd1235"
Oct 30 22:26:10.011544 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:10.010027    5678 scope.go:122] "RemoveContainer" containerID="8afe92c9b204030cfad6f3dc612d17be23f6d48382a88b78b930024f42bd1235"
Oct 30 22:26:10.020535 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:26:10.018020    5678 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8afe92c9b204030cfad6f3dc612d17be23f6d48382a88b78b930024f42bd1235\": container with ID starting with 8afe92c9b204030cfad6f3dc612d17be23f6d48382a88b78b930024f42bd1235 not found: ID does not exist" containerID="8afe92c9b204030cfad6f3dc612d17be23f6d48382a88b78b930024f42bd1235"
Oct 30 22:26:10.020535 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:10.018093    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8afe92c9b204030cfad6f3dc612d17be23f6d48382a88b78b930024f42bd1235"} err="failed to get container status \"8afe92c9b204030cfad6f3dc612d17be23f6d48382a88b78b930024f42bd1235\": rpc error: code = NotFound desc = could not find container \"8afe92c9b204030cfad6f3dc612d17be23f6d48382a88b78b930024f42bd1235\": container with ID starting with 8afe92c9b204030cfad6f3dc612d17be23f6d48382a88b78b930024f42bd1235 not found: ID does not exist"
Oct 30 22:26:10.158170 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:10.158134    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9bfdx"]
Oct 30 22:26:10.163365 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:10.163339    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9bfdx"]
Oct 30 22:26:10.524254 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:10.524210    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b614bbd-09b5-4b79-a9a5-1c57ae795852" path="/var/lib/kubelet/pods/1b614bbd-09b5-4b79-a9a5-1c57ae795852/volumes"
Oct 30 22:26:10.572471 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:26:10.572435    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 22:26:14.735471 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:14.735439    5678 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-9bfdx container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.95:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:26:14.735950 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:14.735491    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9bfdx" podUID="1b614bbd-09b5-4b79-a9a5-1c57ae795852" containerName="discovery" probeResult="failure" output="Get \"http://10.134.0.95:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:26:19.706036 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:19.705996    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-5686cb-ff9ffd898-k5pgw"]
Oct 30 22:26:19.706952 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:19.706922    5678 cpu_manager.go:414] "containerMap: removing container" podUID="09fbaa51-372b-406e-a478-3a7d52a8a618" containerName="istio-proxy"
Oct 30 22:26:19.706952 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:19.706949    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09fbaa51-372b-406e-a478-3a7d52a8a618" containerName="istio-proxy"
Oct 30 22:26:19.707169 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:19.706967    5678 cpu_manager.go:414] "containerMap: removing container" podUID="1b614bbd-09b5-4b79-a9a5-1c57ae795852" containerName="discovery"
Oct 30 22:26:19.707169 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:19.706973    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b614bbd-09b5-4b79-a9a5-1c57ae795852" containerName="discovery"
Oct 30 22:26:19.707290 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:19.707191    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="09fbaa51-372b-406e-a478-3a7d52a8a618" containerName="istio-proxy"
Oct 30 22:26:19.707290 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:19.707207    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b614bbd-09b5-4b79-a9a5-1c57ae795852" containerName="discovery"
Oct 30 22:26:19.711183 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:19.711164    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5686cb-ff9ffd898-k5pgw"
Oct 30 22:26:19.810088 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:19.810054    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-5686cb-serving-cert\""
Oct 30 22:26:19.814116 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:19.814091    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-5686cb-ff9ffd898-k5pgw"]
Oct 30 22:26:19.889732 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:19.889701    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/552ec7ea-ea71-43fa-8129-9829f804fab6-maas-api-tls\") pod \"maas-api-e2e-api-b-5686cb-ff9ffd898-k5pgw\" (UID: \"552ec7ea-ea71-43fa-8129-9829f804fab6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5686cb-ff9ffd898-k5pgw"
Oct 30 22:26:19.889955 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:19.889932    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/552ec7ea-ea71-43fa-8129-9829f804fab6-metrics-certs\") pod \"maas-api-e2e-api-b-5686cb-ff9ffd898-k5pgw\" (UID: \"552ec7ea-ea71-43fa-8129-9829f804fab6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5686cb-ff9ffd898-k5pgw"
Oct 30 22:26:19.890064 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:19.890045    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zslp\" (UniqueName: \"kubernetes.io/projected/552ec7ea-ea71-43fa-8129-9829f804fab6-kube-api-access-4zslp\") pod \"maas-api-e2e-api-b-5686cb-ff9ffd898-k5pgw\" (UID: \"552ec7ea-ea71-43fa-8129-9829f804fab6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5686cb-ff9ffd898-k5pgw"
Oct 30 22:26:19.890136 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:19.890122    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/552ec7ea-ea71-43fa-8129-9829f804fab6-openshift-service-ca\") pod \"maas-api-e2e-api-b-5686cb-ff9ffd898-k5pgw\" (UID: \"552ec7ea-ea71-43fa-8129-9829f804fab6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5686cb-ff9ffd898-k5pgw"
Oct 30 22:26:19.998785 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:19.998555    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/552ec7ea-ea71-43fa-8129-9829f804fab6-metrics-certs\") pod \"maas-api-e2e-api-b-5686cb-ff9ffd898-k5pgw\" (UID: \"552ec7ea-ea71-43fa-8129-9829f804fab6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5686cb-ff9ffd898-k5pgw"
Oct 30 22:26:19.998785 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:19.998656    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zslp\" (UniqueName: \"kubernetes.io/projected/552ec7ea-ea71-43fa-8129-9829f804fab6-kube-api-access-4zslp\") pod \"maas-api-e2e-api-b-5686cb-ff9ffd898-k5pgw\" (UID: \"552ec7ea-ea71-43fa-8129-9829f804fab6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5686cb-ff9ffd898-k5pgw"
Oct 30 22:26:19.998785 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:19.998723    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/552ec7ea-ea71-43fa-8129-9829f804fab6-openshift-service-ca\") pod \"maas-api-e2e-api-b-5686cb-ff9ffd898-k5pgw\" (UID: \"552ec7ea-ea71-43fa-8129-9829f804fab6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5686cb-ff9ffd898-k5pgw"
Oct 30 22:26:19.999108 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:19.998826    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/552ec7ea-ea71-43fa-8129-9829f804fab6-maas-api-tls\") pod \"maas-api-e2e-api-b-5686cb-ff9ffd898-k5pgw\" (UID: \"552ec7ea-ea71-43fa-8129-9829f804fab6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5686cb-ff9ffd898-k5pgw"
Oct 30 22:26:20.005190 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:20.005132    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/552ec7ea-ea71-43fa-8129-9829f804fab6-openshift-service-ca\") pod \"maas-api-e2e-api-b-5686cb-ff9ffd898-k5pgw\" (UID: \"552ec7ea-ea71-43fa-8129-9829f804fab6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5686cb-ff9ffd898-k5pgw"
Oct 30 22:26:20.013819 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:20.013758    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/552ec7ea-ea71-43fa-8129-9829f804fab6-metrics-certs\") pod \"maas-api-e2e-api-b-5686cb-ff9ffd898-k5pgw\" (UID: \"552ec7ea-ea71-43fa-8129-9829f804fab6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5686cb-ff9ffd898-k5pgw"
Oct 30 22:26:20.016180 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:20.016144    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/552ec7ea-ea71-43fa-8129-9829f804fab6-maas-api-tls\") pod \"maas-api-e2e-api-b-5686cb-ff9ffd898-k5pgw\" (UID: \"552ec7ea-ea71-43fa-8129-9829f804fab6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5686cb-ff9ffd898-k5pgw"
Oct 30 22:26:20.057885 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:20.057855    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zslp\" (UniqueName: \"kubernetes.io/projected/552ec7ea-ea71-43fa-8129-9829f804fab6-kube-api-access-4zslp\") pod \"maas-api-e2e-api-b-5686cb-ff9ffd898-k5pgw\" (UID: \"552ec7ea-ea71-43fa-8129-9829f804fab6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5686cb-ff9ffd898-k5pgw"
Oct 30 22:26:20.325208 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:20.325128    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5686cb-ff9ffd898-k5pgw"
Oct 30 22:26:20.720540 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:20.720509    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-5686cb-ff9ffd898-k5pgw"]
Oct 30 22:26:20.725839 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:26:20.725814    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod552ec7ea_ea71_43fa_8129_9829f804fab6.slice/crio-c3ef68c56f5cff102743d5fb864d4b082b237b75f3d414d747d80c3a672cf63e WatchSource:0}: Error finding container c3ef68c56f5cff102743d5fb864d4b082b237b75f3d414d747d80c3a672cf63e: Status 404 returned error can't find the container with id c3ef68c56f5cff102743d5fb864d4b082b237b75f3d414d747d80c3a672cf63e
Oct 30 22:26:21.088199 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:21.088117    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5686cb-ff9ffd898-k5pgw" event={"ID":"552ec7ea-ea71-43fa-8129-9829f804fab6","Type":"ContainerStarted","Data":"c3ef68c56f5cff102743d5fb864d4b082b237b75f3d414d747d80c3a672cf63e"}
Oct 30 22:26:22.097589 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:22.097550    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5686cb-ff9ffd898-k5pgw" event={"ID":"552ec7ea-ea71-43fa-8129-9829f804fab6","Type":"ContainerStarted","Data":"d27d2ea771ec8e2db7943ef34f4faf9678ebded549da1747a28daf41658e5222"}
Oct 30 22:26:22.097978 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:22.097702    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5686cb-ff9ffd898-k5pgw"
Oct 30 22:26:22.215678 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:22.215624    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5686cb-ff9ffd898-k5pgw" podStartSLOduration=2.365526586 podStartE2EDuration="3.215608541s" totalImagesPullingTime="850.081955ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:26:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:26:22.212933894 +0000 UTC m=+2159.817241497" watchObservedRunningTime="2026-10-30 22:26:22.215608541 +0000 UTC m=+2159.819916095"
Oct 30 22:26:28.174227 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:28.172859    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5686cb-ff9ffd898-k5pgw"
Oct 30 22:26:31.812609 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:31.812558    5678 patch_prober.go:29] interesting pod/e2e-inf-a-5637a7-openshift-default-5b47575485-dxr9g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:26:31.812997 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:31.812650    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-5637a7-openshift-default-5b47575485-dxr9g" podUID="9c5d0357-a8f0-484c-9584-4b0184fd5284" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:26:35.253094 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:35.253013    5678 generic.go:356] "Generic (PLEG): container finished" podID="9c5d0357-a8f0-484c-9584-4b0184fd5284" containerID="79480cc4e93133cbe7c6c8aa113e4499d86701e9e32b9d50abb75fa7606c27af" exitCode=0
Oct 30 22:26:35.253594 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:35.253085    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5637a7-openshift-default-5b47575485-dxr9g" event={"ID":"9c5d0357-a8f0-484c-9584-4b0184fd5284","Type":"ContainerDied","Data":"79480cc4e93133cbe7c6c8aa113e4499d86701e9e32b9d50abb75fa7606c27af"}
Oct 30 22:26:35.253685 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:35.253595    5678 scope.go:122] "RemoveContainer" containerID="79480cc4e93133cbe7c6c8aa113e4499d86701e9e32b9d50abb75fa7606c27af"
Oct 30 22:26:35.254855 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:35.254832    5678 generic.go:356] "Generic (PLEG): container finished" podID="6f9052f2-a2a5-47a5-99b8-138d4e59755c" containerID="2bb759a4dd66467471b7996985692e19c6c378e305e2e6d052dcdf00bd1d7daf" exitCode=0
Oct 30 22:26:35.254945 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:35.254870    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-9f3ea1-openshift-default-5b767dc779-rbqjf" event={"ID":"6f9052f2-a2a5-47a5-99b8-138d4e59755c","Type":"ContainerDied","Data":"2bb759a4dd66467471b7996985692e19c6c378e305e2e6d052dcdf00bd1d7daf"}
Oct 30 22:26:35.255199 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:35.255186    5678 scope.go:122] "RemoveContainer" containerID="2bb759a4dd66467471b7996985692e19c6c378e305e2e6d052dcdf00bd1d7daf"
Oct 30 22:26:35.255235 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:35.255189    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:26:35.255272 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:35.255256    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:26:35.255310 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:35.255289    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:26:35.256628 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:35.256603    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:26:35.256715 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:35.256664    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:26:35.256715 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:35.256698    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:26:35.814445 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:35.814415    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-9f3ea1-openshift-default-5b767dc779-rbqjf"
Oct 30 22:26:36.283228 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:36.283192    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5637a7-openshift-default-5b47575485-dxr9g" event={"ID":"9c5d0357-a8f0-484c-9584-4b0184fd5284","Type":"ContainerStarted","Data":"8a0e4e964afec05f0364932bdf0add52bb8c1557985001aa137064730590af54"}
Oct 30 22:26:36.285470 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:36.285446    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-9f3ea1-openshift-default-5b767dc779-rbqjf" event={"ID":"6f9052f2-a2a5-47a5-99b8-138d4e59755c","Type":"ContainerStarted","Data":"f71fdb8fcdf8914f1596b4429501252507c089c3b7623700dbad9f067780248b"}
Oct 30 22:26:36.366304 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:36.366267    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-5637a7-openshift-default-5b47575485-dxr9g"
Oct 30 22:26:36.373679 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:36.367683    5678 patch_prober.go:29] interesting pod/e2e-inf-a-5637a7-openshift-default-5b47575485-dxr9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body=
Oct 30 22:26:36.373896 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:36.373869    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5637a7-openshift-default-5b47575485-dxr9g" podUID="9c5d0357-a8f0-484c-9584-4b0184fd5284" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused"
Oct 30 22:26:36.675268 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:36.675221    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-9f3ea1-openshift-default-5b767dc779-rbqjf"
Oct 30 22:26:36.684556 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:36.684529    5678 patch_prober.go:29] interesting pod/e2e-shared-b-w2-9f3ea1-openshift-default-5b767dc779-rbqjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body=
Oct 30 22:26:36.684761 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:36.684740    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-9f3ea1-openshift-default-5b767dc779-rbqjf" podUID="6f9052f2-a2a5-47a5-99b8-138d4e59755c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused"
Oct 30 22:26:37.351425 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:37.342755    5678 patch_prober.go:29] interesting pod/e2e-inf-a-5637a7-openshift-default-5b47575485-dxr9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body=
Oct 30 22:26:37.351425 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:37.342816    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5637a7-openshift-default-5b47575485-dxr9g" podUID="9c5d0357-a8f0-484c-9584-4b0184fd5284" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused"
Oct 30 22:26:37.674853 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:37.674814    5678 patch_prober.go:29] interesting pod/e2e-shared-b-w2-9f3ea1-openshift-default-5b767dc779-rbqjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body=
Oct 30 22:26:37.675068 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:37.674869    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-9f3ea1-openshift-default-5b767dc779-rbqjf" podUID="6f9052f2-a2a5-47a5-99b8-138d4e59755c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused"
Oct 30 22:26:38.337821 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:38.337782    5678 patch_prober.go:29] interesting pod/e2e-inf-a-5637a7-openshift-default-5b47575485-dxr9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body=
Oct 30 22:26:38.338129 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:38.338099    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5637a7-openshift-default-5b47575485-dxr9g" podUID="9c5d0357-a8f0-484c-9584-4b0184fd5284" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused"
Oct 30 22:26:38.677426 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:38.676746    5678 patch_prober.go:29] interesting pod/e2e-shared-b-w2-9f3ea1-openshift-default-5b767dc779-rbqjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body=
Oct 30 22:26:38.677426 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:38.676805    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-9f3ea1-openshift-default-5b767dc779-rbqjf" podUID="6f9052f2-a2a5-47a5-99b8-138d4e59755c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused"
Oct 30 22:26:38.677426 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:38.677358    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c24d61-5b9ccff96c-klgks"]
Oct 30 22:26:38.773493 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:38.773436    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c24d61-5b86d448c-69ql5"]
Oct 30 22:26:39.349429 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:39.347715    5678 patch_prober.go:29] interesting pod/e2e-inf-a-5637a7-openshift-default-5b47575485-dxr9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body=
Oct 30 22:26:39.349429 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:39.347794    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5637a7-openshift-default-5b47575485-dxr9g" podUID="9c5d0357-a8f0-484c-9584-4b0184fd5284" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused"
Oct 30 22:26:39.675557 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:39.675520    5678 patch_prober.go:29] interesting pod/e2e-shared-b-w2-9f3ea1-openshift-default-5b767dc779-rbqjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body=
Oct 30 22:26:39.675870 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:39.675840    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-9f3ea1-openshift-default-5b767dc779-rbqjf" podUID="6f9052f2-a2a5-47a5-99b8-138d4e59755c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused"
Oct 30 22:26:40.348788 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:40.348753    5678 patch_prober.go:29] interesting pod/e2e-inf-a-5637a7-openshift-default-5b47575485-dxr9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body=
Oct 30 22:26:40.349442 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:40.349388    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5637a7-openshift-default-5b47575485-dxr9g" podUID="9c5d0357-a8f0-484c-9584-4b0184fd5284" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused"
Oct 30 22:26:40.703852 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:40.703813    5678 patch_prober.go:29] interesting pod/e2e-shared-b-w2-9f3ea1-openshift-default-5b767dc779-rbqjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body=
Oct 30 22:26:40.704142 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:40.704111    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-9f3ea1-openshift-default-5b767dc779-rbqjf" podUID="6f9052f2-a2a5-47a5-99b8-138d4e59755c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused"
Oct 30 22:26:40.841471 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:26:40.838704    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 22:26:41.343827 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:41.343790    5678 patch_prober.go:29] interesting pod/e2e-inf-a-5637a7-openshift-default-5b47575485-dxr9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body=
Oct 30 22:26:41.344144 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:41.344114    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5637a7-openshift-default-5b47575485-dxr9g" podUID="9c5d0357-a8f0-484c-9584-4b0184fd5284" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused"
Oct 30 22:26:41.675512 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:41.675475    5678 patch_prober.go:29] interesting pod/e2e-shared-b-w2-9f3ea1-openshift-default-5b767dc779-rbqjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body=
Oct 30 22:26:41.676531 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:41.676492    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-9f3ea1-openshift-default-5b767dc779-rbqjf" podUID="6f9052f2-a2a5-47a5-99b8-138d4e59755c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused"
Oct 30 22:26:42.346695 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:42.346662    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-5637a7-openshift-default-5b47575485-dxr9g"
Oct 30 22:26:42.348031 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:42.347998    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-5637a7-openshift-default-5b47575485-dxr9g"
Oct 30 22:26:42.350600 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:42.350575    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-5637a7-openshift-default-5b47575485-dxr9g"
Oct 30 22:26:42.680778 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:42.680742    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-9f3ea1-openshift-default-5b767dc779-rbqjf"
Oct 30 22:26:42.681619 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:42.681590    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-9f3ea1-openshift-default-5b767dc779-rbqjf"
Oct 30 22:26:42.682119 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:42.682101    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-9f3ea1-openshift-default-5b767dc779-rbqjf"
Oct 30 22:26:43.677660 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:43.677618    5678 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-c24d61-5b9ccff96c-klgks" podUID="9c1a81bf-5ffe-4f8a-9295-781400c2f261" containerName="payload-processing" containerID="cri-o://88f74be73331ce950de36b8cb82a5d458d93e0aa42bd9155394972ff9c2c8859" gracePeriod=25
Oct 30 22:26:43.774631 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:43.774594    5678 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c24d61-5b86d448c-69ql5" podUID="cafd6d42-da8f-4b65-bcfa-41e67dc3b90e" containerName="payload-pre-processing" containerID="cri-o://e4cd15fd1503db611a73f1d3f1cefe2cd71eb4255e12f57a37f9e6b9a096fa31" gracePeriod=25
Oct 30 22:26:44.067902 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:44.067860    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c24d61-5b9ccff96c-klgks"
Oct 30 22:26:44.144422 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:44.143696    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c1a81bf-5ffe-4f8a-9295-781400c2f261-kube-api-access-7g92w\" (UniqueName: \"kubernetes.io/projected/9c1a81bf-5ffe-4f8a-9295-781400c2f261-kube-api-access-7g92w\") pod \"9c1a81bf-5ffe-4f8a-9295-781400c2f261\" (UID: \"9c1a81bf-5ffe-4f8a-9295-781400c2f261\") "
Oct 30 22:26:44.144422 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:44.143934    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c1a81bf-5ffe-4f8a-9295-781400c2f261-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c1a81bf-5ffe-4f8a-9295-781400c2f261-plugins-config-volume\") pod \"9c1a81bf-5ffe-4f8a-9295-781400c2f261\" (UID: \"9c1a81bf-5ffe-4f8a-9295-781400c2f261\") "
Oct 30 22:26:44.157909 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:44.157009    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c1a81bf-5ffe-4f8a-9295-781400c2f261-plugins-config-volume" pod "9c1a81bf-5ffe-4f8a-9295-781400c2f261" (UID: "9c1a81bf-5ffe-4f8a-9295-781400c2f261"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:26:44.161389 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:44.160916    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c1a81bf-5ffe-4f8a-9295-781400c2f261-kube-api-access-7g92w" pod "9c1a81bf-5ffe-4f8a-9295-781400c2f261" (UID: "9c1a81bf-5ffe-4f8a-9295-781400c2f261"). InnerVolumeSpecName "kube-api-access-7g92w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:26:44.191678 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:44.191652    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c24d61-5b86d448c-69ql5"
Oct 30 22:26:44.249680 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:44.248367    5678 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c1a81bf-5ffe-4f8a-9295-781400c2f261-plugins-config-volume\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:26:44.249680 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:44.248434    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7g92w\" (UniqueName: \"kubernetes.io/projected/9c1a81bf-5ffe-4f8a-9295-781400c2f261-kube-api-access-7g92w\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:26:44.351984 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:44.351946    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cafd6d42-da8f-4b65-bcfa-41e67dc3b90e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cafd6d42-da8f-4b65-bcfa-41e67dc3b90e-plugins-config-volume\") pod \"cafd6d42-da8f-4b65-bcfa-41e67dc3b90e\" (UID: \"cafd6d42-da8f-4b65-bcfa-41e67dc3b90e\") "
Oct 30 22:26:44.352158 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:44.352016    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cafd6d42-da8f-4b65-bcfa-41e67dc3b90e-kube-api-access-cw588\" (UniqueName: \"kubernetes.io/projected/cafd6d42-da8f-4b65-bcfa-41e67dc3b90e-kube-api-access-cw588\") pod \"cafd6d42-da8f-4b65-bcfa-41e67dc3b90e\" (UID: \"cafd6d42-da8f-4b65-bcfa-41e67dc3b90e\") "
Oct 30 22:26:44.352470 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:44.352439    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cafd6d42-da8f-4b65-bcfa-41e67dc3b90e-plugins-config-volume" pod "cafd6d42-da8f-4b65-bcfa-41e67dc3b90e" (UID: "cafd6d42-da8f-4b65-bcfa-41e67dc3b90e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:26:44.352644 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:44.352623    5678 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cafd6d42-da8f-4b65-bcfa-41e67dc3b90e-plugins-config-volume\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:26:44.355716 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:44.355688    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cafd6d42-da8f-4b65-bcfa-41e67dc3b90e-kube-api-access-cw588" pod "cafd6d42-da8f-4b65-bcfa-41e67dc3b90e" (UID: "cafd6d42-da8f-4b65-bcfa-41e67dc3b90e"). InnerVolumeSpecName "kube-api-access-cw588". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:26:44.453934 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:44.453891    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cw588\" (UniqueName: \"kubernetes.io/projected/cafd6d42-da8f-4b65-bcfa-41e67dc3b90e-kube-api-access-cw588\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:26:44.514988 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:44.514904    5678 generic.go:356] "Generic (PLEG): container finished" podID="9c1a81bf-5ffe-4f8a-9295-781400c2f261" containerID="88f74be73331ce950de36b8cb82a5d458d93e0aa42bd9155394972ff9c2c8859" exitCode=0
Oct 30 22:26:44.514988 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:44.514956    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c24d61-5b9ccff96c-klgks" event={"ID":"9c1a81bf-5ffe-4f8a-9295-781400c2f261","Type":"ContainerDied","Data":"88f74be73331ce950de36b8cb82a5d458d93e0aa42bd9155394972ff9c2c8859"}
Oct 30 22:26:44.515203 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:44.515036    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c24d61-5b9ccff96c-klgks" event={"ID":"9c1a81bf-5ffe-4f8a-9295-781400c2f261","Type":"ContainerDied","Data":"1f668ee27f95a34e851d8f27bb82cf4a9c809c5a6522e19c320cbcfd20af1e70"}
Oct 30 22:26:44.515203 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:44.514964    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c24d61-5b9ccff96c-klgks"
Oct 30 22:26:44.515203 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:44.515064    5678 scope.go:122] "RemoveContainer" containerID="88f74be73331ce950de36b8cb82a5d458d93e0aa42bd9155394972ff9c2c8859"
Oct 30 22:26:44.517254 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:44.517226    5678 generic.go:356] "Generic (PLEG): container finished" podID="cafd6d42-da8f-4b65-bcfa-41e67dc3b90e" containerID="e4cd15fd1503db611a73f1d3f1cefe2cd71eb4255e12f57a37f9e6b9a096fa31" exitCode=0
Oct 30 22:26:44.517395 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:44.517327    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c24d61-5b86d448c-69ql5"
Oct 30 22:26:44.552841 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:44.552807    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c24d61-5b86d448c-69ql5" event={"ID":"cafd6d42-da8f-4b65-bcfa-41e67dc3b90e","Type":"ContainerDied","Data":"e4cd15fd1503db611a73f1d3f1cefe2cd71eb4255e12f57a37f9e6b9a096fa31"}
Oct 30 22:26:44.552984 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:44.552864    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c24d61-5b86d448c-69ql5" event={"ID":"cafd6d42-da8f-4b65-bcfa-41e67dc3b90e","Type":"ContainerDied","Data":"bf30c433d9d164c313d6b69f7c7747a555030315d633e78b42e8817052ea812d"}
Oct 30 22:26:44.565700 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:44.565674    5678 scope.go:122] "RemoveContainer" containerID="88f74be73331ce950de36b8cb82a5d458d93e0aa42bd9155394972ff9c2c8859"
Oct 30 22:26:44.566701 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:26:44.566668    5678 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88f74be73331ce950de36b8cb82a5d458d93e0aa42bd9155394972ff9c2c8859\": container with ID starting with 88f74be73331ce950de36b8cb82a5d458d93e0aa42bd9155394972ff9c2c8859 not found: ID does not exist" containerID="88f74be73331ce950de36b8cb82a5d458d93e0aa42bd9155394972ff9c2c8859"
Oct 30 22:26:44.566842 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:44.566719    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88f74be73331ce950de36b8cb82a5d458d93e0aa42bd9155394972ff9c2c8859"} err="failed to get container status \"88f74be73331ce950de36b8cb82a5d458d93e0aa42bd9155394972ff9c2c8859\": rpc error: code = NotFound desc = could not find container \"88f74be73331ce950de36b8cb82a5d458d93e0aa42bd9155394972ff9c2c8859\": container with ID starting with 88f74be73331ce950de36b8cb82a5d458d93e0aa42bd9155394972ff9c2c8859 not found: ID does not exist"
Oct 30 22:26:44.566842 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:44.566742    5678 scope.go:122] "RemoveContainer" containerID="e4cd15fd1503db611a73f1d3f1cefe2cd71eb4255e12f57a37f9e6b9a096fa31"
Oct 30 22:26:44.576326 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:44.576308    5678 scope.go:122] "RemoveContainer" containerID="e4cd15fd1503db611a73f1d3f1cefe2cd71eb4255e12f57a37f9e6b9a096fa31"
Oct 30 22:26:44.576627 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:26:44.576607    5678 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4cd15fd1503db611a73f1d3f1cefe2cd71eb4255e12f57a37f9e6b9a096fa31\": container with ID starting with e4cd15fd1503db611a73f1d3f1cefe2cd71eb4255e12f57a37f9e6b9a096fa31 not found: ID does not exist" containerID="e4cd15fd1503db611a73f1d3f1cefe2cd71eb4255e12f57a37f9e6b9a096fa31"
Oct 30 22:26:44.576704 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:44.576637    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4cd15fd1503db611a73f1d3f1cefe2cd71eb4255e12f57a37f9e6b9a096fa31"} err="failed to get container status \"e4cd15fd1503db611a73f1d3f1cefe2cd71eb4255e12f57a37f9e6b9a096fa31\": rpc error: code = NotFound desc = could not find container \"e4cd15fd1503db611a73f1d3f1cefe2cd71eb4255e12f57a37f9e6b9a096fa31\": container with ID starting with e4cd15fd1503db611a73f1d3f1cefe2cd71eb4255e12f57a37f9e6b9a096fa31 not found: ID does not exist"
Oct 30 22:26:44.675299 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:44.675250    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c24d61-5b86d448c-69ql5"]
Oct 30 22:26:44.680127 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:44.680103    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c24d61-5b86d448c-69ql5"]
Oct 30 22:26:44.784816 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:44.784726    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c24d61-5b9ccff96c-klgks"]
Oct 30 22:26:44.789730 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:44.789701    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c24d61-5b9ccff96c-klgks"]
Oct 30 22:26:45.050423 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:45.050319    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d143f49-swg9z"]
Oct 30 22:26:45.054673 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:45.054646    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d143f49-swg9z"]
Oct 30 22:26:46.526563 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:46.526527    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c1a81bf-5ffe-4f8a-9295-781400c2f261" path="/var/lib/kubelet/pods/9c1a81bf-5ffe-4f8a-9295-781400c2f261/volumes"
Oct 30 22:26:46.527145 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:46.527129    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cafd6d42-da8f-4b65-bcfa-41e67dc3b90e" path="/var/lib/kubelet/pods/cafd6d42-da8f-4b65-bcfa-41e67dc3b90e/volumes"
Oct 30 22:26:46.527640 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:46.527624    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee597b98-9681-40c7-8af6-5484942d8094" path="/var/lib/kubelet/pods/ee597b98-9681-40c7-8af6-5484942d8094/volumes"
Oct 30 22:26:48.548036 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:48.547942    5678 generic.go:356] "Generic (PLEG): container finished" podID="bd8986aa-b616-4a9e-9dd9-ae7eefd3e847" containerID="8bed24e326a634e2ce195cb92ff0d418033527731d0cb1f61ca976b2d01ce2af" exitCode=0
Oct 30 22:26:48.548036 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:48.548014    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c24d61-openshift-default-7c56bd489-c85l8" event={"ID":"bd8986aa-b616-4a9e-9dd9-ae7eefd3e847","Type":"ContainerDied","Data":"8bed24e326a634e2ce195cb92ff0d418033527731d0cb1f61ca976b2d01ce2af"}
Oct 30 22:26:48.548507 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:48.548365    5678 scope.go:122] "RemoveContainer" containerID="8bed24e326a634e2ce195cb92ff0d418033527731d0cb1f61ca976b2d01ce2af"
Oct 30 22:26:48.550133 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:48.550097    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:26:48.550248 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:48.550180    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:26:48.550248 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:48.550226    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:26:49.557869 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:49.557835    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c24d61-openshift-default-7c56bd489-c85l8" event={"ID":"bd8986aa-b616-4a9e-9dd9-ae7eefd3e847","Type":"ContainerStarted","Data":"69ca4fb4a4f0d20bb76fac59352f175389dd3e67b6534374f201b30d9c3999ba"}
Oct 30 22:26:50.052773 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:50.052733    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-c24d61-openshift-default-7c56bd489-c85l8"
Oct 30 22:26:50.052982 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:50.052915    5678 patch_prober.go:29] interesting pod/e2e-api-a-c24d61-openshift-default-7c56bd489-c85l8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body=
Oct 30 22:26:50.053039 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:50.052978    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c24d61-openshift-default-7c56bd489-c85l8" podUID="bd8986aa-b616-4a9e-9dd9-ae7eefd3e847" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused"
Oct 30 22:26:51.053553 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:51.053503    5678 patch_prober.go:29] interesting pod/e2e-api-a-c24d61-openshift-default-7c56bd489-c85l8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body=
Oct 30 22:26:51.054081 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:51.053580    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c24d61-openshift-default-7c56bd489-c85l8" podUID="bd8986aa-b616-4a9e-9dd9-ae7eefd3e847" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused"
Oct 30 22:26:52.053121 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:52.053081    5678 patch_prober.go:29] interesting pod/e2e-api-a-c24d61-openshift-default-7c56bd489-c85l8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body=
Oct 30 22:26:52.053317 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:52.053149    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c24d61-openshift-default-7c56bd489-c85l8" podUID="bd8986aa-b616-4a9e-9dd9-ae7eefd3e847" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused"
Oct 30 22:26:53.061811 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:53.060425    5678 patch_prober.go:29] interesting pod/e2e-api-a-c24d61-openshift-default-7c56bd489-c85l8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body=
Oct 30 22:26:53.061811 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:53.060586    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c24d61-openshift-default-7c56bd489-c85l8" podUID="bd8986aa-b616-4a9e-9dd9-ae7eefd3e847" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused"
Oct 30 22:26:54.052934 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:54.052897    5678 patch_prober.go:29] interesting pod/e2e-api-a-c24d61-openshift-default-7c56bd489-c85l8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body=
Oct 30 22:26:54.053121 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:54.052960    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c24d61-openshift-default-7c56bd489-c85l8" podUID="bd8986aa-b616-4a9e-9dd9-ae7eefd3e847" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused"
Oct 30 22:26:55.076152 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:55.076120    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-c24d61-openshift-default-7c56bd489-c85l8"
Oct 30 22:26:55.076699 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:55.076683    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-c24d61-openshift-default-7c56bd489-c85l8"
Oct 30 22:26:55.077302 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:26:55.077285    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-c24d61-openshift-default-7c56bd489-c85l8"
Oct 30 22:27:01.792675 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:01.792644    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d7cdcc-c769d69c6-7z9lp"]
Oct 30 22:27:01.793313 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:01.793297    5678 cpu_manager.go:414] "containerMap: removing container" podUID="9c1a81bf-5ffe-4f8a-9295-781400c2f261" containerName="payload-processing"
Oct 30 22:27:01.793359 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:01.793315    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c1a81bf-5ffe-4f8a-9295-781400c2f261" containerName="payload-processing"
Oct 30 22:27:01.793359 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:01.793339    5678 cpu_manager.go:414] "containerMap: removing container" podUID="cafd6d42-da8f-4b65-bcfa-41e67dc3b90e" containerName="payload-pre-processing"
Oct 30 22:27:01.793359 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:01.793347    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cafd6d42-da8f-4b65-bcfa-41e67dc3b90e" containerName="payload-pre-processing"
Oct 30 22:27:01.793517 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:01.793505    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c1a81bf-5ffe-4f8a-9295-781400c2f261" containerName="payload-processing"
Oct 30 22:27:01.793552 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:01.793523    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="cafd6d42-da8f-4b65-bcfa-41e67dc3b90e" containerName="payload-pre-processing"
Oct 30 22:27:01.796860 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:01.796840    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d7cdcc-c769d69c6-7z9lp"
Oct 30 22:27:01.858901 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:01.858860    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1686023a-bf9e-4224-82ea-1c72bc4273c6-openshift-service-ca\") pod \"maas-api-e2e-inf-a-d7cdcc-c769d69c6-7z9lp\" (UID: \"1686023a-bf9e-4224-82ea-1c72bc4273c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d7cdcc-c769d69c6-7z9lp"
Oct 30 22:27:01.859093 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:01.858938    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1686023a-bf9e-4224-82ea-1c72bc4273c6-metrics-certs\") pod \"maas-api-e2e-inf-a-d7cdcc-c769d69c6-7z9lp\" (UID: \"1686023a-bf9e-4224-82ea-1c72bc4273c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d7cdcc-c769d69c6-7z9lp"
Oct 30 22:27:01.859093 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:01.859089    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1686023a-bf9e-4224-82ea-1c72bc4273c6-maas-api-tls\") pod \"maas-api-e2e-inf-a-d7cdcc-c769d69c6-7z9lp\" (UID: \"1686023a-bf9e-4224-82ea-1c72bc4273c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d7cdcc-c769d69c6-7z9lp"
Oct 30 22:27:01.859221 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:01.859158    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8qlm\" (UniqueName: \"kubernetes.io/projected/1686023a-bf9e-4224-82ea-1c72bc4273c6-kube-api-access-t8qlm\") pod \"maas-api-e2e-inf-a-d7cdcc-c769d69c6-7z9lp\" (UID: \"1686023a-bf9e-4224-82ea-1c72bc4273c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d7cdcc-c769d69c6-7z9lp"
Oct 30 22:27:01.894658 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:01.894632    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-d7cdcc-serving-cert\""
Oct 30 22:27:01.898735 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:01.898714    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d7cdcc-c769d69c6-7z9lp"]
Oct 30 22:27:01.960299 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:01.960263    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1686023a-bf9e-4224-82ea-1c72bc4273c6-maas-api-tls\") pod \"maas-api-e2e-inf-a-d7cdcc-c769d69c6-7z9lp\" (UID: \"1686023a-bf9e-4224-82ea-1c72bc4273c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d7cdcc-c769d69c6-7z9lp"
Oct 30 22:27:01.960503 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:01.960331    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8qlm\" (UniqueName: \"kubernetes.io/projected/1686023a-bf9e-4224-82ea-1c72bc4273c6-kube-api-access-t8qlm\") pod \"maas-api-e2e-inf-a-d7cdcc-c769d69c6-7z9lp\" (UID: \"1686023a-bf9e-4224-82ea-1c72bc4273c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d7cdcc-c769d69c6-7z9lp"
Oct 30 22:27:01.960503 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:01.960393    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1686023a-bf9e-4224-82ea-1c72bc4273c6-openshift-service-ca\") pod \"maas-api-e2e-inf-a-d7cdcc-c769d69c6-7z9lp\" (UID: \"1686023a-bf9e-4224-82ea-1c72bc4273c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d7cdcc-c769d69c6-7z9lp"
Oct 30 22:27:01.960503 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:27:01.960442    5678 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-d7cdcc-serving-cert: secret "maas-api-e2e-inf-a-d7cdcc-serving-cert" not found
Oct 30 22:27:01.960670 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:01.960447    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1686023a-bf9e-4224-82ea-1c72bc4273c6-metrics-certs\") pod \"maas-api-e2e-inf-a-d7cdcc-c769d69c6-7z9lp\" (UID: \"1686023a-bf9e-4224-82ea-1c72bc4273c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d7cdcc-c769d69c6-7z9lp"
Oct 30 22:27:01.960670 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:27:01.960529    5678 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1686023a-bf9e-4224-82ea-1c72bc4273c6-maas-api-tls podName:1686023a-bf9e-4224-82ea-1c72bc4273c6 nodeName:}" failed. No retries permitted until 2026-10-30 22:27:02.460508355 +0000 UTC m=+2200.064815909 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1686023a-bf9e-4224-82ea-1c72bc4273c6-maas-api-tls") pod "maas-api-e2e-inf-a-d7cdcc-c769d69c6-7z9lp" (UID: "1686023a-bf9e-4224-82ea-1c72bc4273c6") : secret "maas-api-e2e-inf-a-d7cdcc-serving-cert" not found
Oct 30 22:27:01.961079 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:01.961054    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1686023a-bf9e-4224-82ea-1c72bc4273c6-openshift-service-ca\") pod \"maas-api-e2e-inf-a-d7cdcc-c769d69c6-7z9lp\" (UID: \"1686023a-bf9e-4224-82ea-1c72bc4273c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d7cdcc-c769d69c6-7z9lp"
Oct 30 22:27:01.962780 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:01.962762    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1686023a-bf9e-4224-82ea-1c72bc4273c6-metrics-certs\") pod \"maas-api-e2e-inf-a-d7cdcc-c769d69c6-7z9lp\" (UID: \"1686023a-bf9e-4224-82ea-1c72bc4273c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d7cdcc-c769d69c6-7z9lp"
Oct 30 22:27:02.058371 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:02.058229    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8qlm\" (UniqueName: \"kubernetes.io/projected/1686023a-bf9e-4224-82ea-1c72bc4273c6-kube-api-access-t8qlm\") pod \"maas-api-e2e-inf-a-d7cdcc-c769d69c6-7z9lp\" (UID: \"1686023a-bf9e-4224-82ea-1c72bc4273c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d7cdcc-c769d69c6-7z9lp"
Oct 30 22:27:02.469674 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:02.469640    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1686023a-bf9e-4224-82ea-1c72bc4273c6-maas-api-tls\") pod \"maas-api-e2e-inf-a-d7cdcc-c769d69c6-7z9lp\" (UID: \"1686023a-bf9e-4224-82ea-1c72bc4273c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d7cdcc-c769d69c6-7z9lp"
Oct 30 22:27:02.472020 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:02.471985    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1686023a-bf9e-4224-82ea-1c72bc4273c6-maas-api-tls\") pod \"maas-api-e2e-inf-a-d7cdcc-c769d69c6-7z9lp\" (UID: \"1686023a-bf9e-4224-82ea-1c72bc4273c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d7cdcc-c769d69c6-7z9lp"
Oct 30 22:27:02.709659 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:02.709629    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d7cdcc-c769d69c6-7z9lp"
Oct 30 22:27:03.099091 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:03.099008    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d7cdcc-c769d69c6-7z9lp"]
Oct 30 22:27:03.104552 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:27:03.104521    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1686023a_bf9e_4224_82ea_1c72bc4273c6.slice/crio-06d733c2d2ad6bf3200f7d4ecdcfc1da98a7b18be77e7ab771ee48ab1b4872db WatchSource:0}: Error finding container 06d733c2d2ad6bf3200f7d4ecdcfc1da98a7b18be77e7ab771ee48ab1b4872db: Status 404 returned error can't find the container with id 06d733c2d2ad6bf3200f7d4ecdcfc1da98a7b18be77e7ab771ee48ab1b4872db
Oct 30 22:27:03.662723 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:03.662688    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d7cdcc-c769d69c6-7z9lp" event={"ID":"1686023a-bf9e-4224-82ea-1c72bc4273c6","Type":"ContainerStarted","Data":"06d733c2d2ad6bf3200f7d4ecdcfc1da98a7b18be77e7ab771ee48ab1b4872db"}
Oct 30 22:27:04.070430 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:04.070388    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dafc795b-hrsc4"]
Oct 30 22:27:04.118082 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:04.118053    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dafc795b-hrsc4"]
Oct 30 22:27:04.524160 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:04.524127    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4845576-736c-471b-83f1-ca535529773d" path="/var/lib/kubelet/pods/e4845576-736c-471b-83f1-ca535529773d/volumes"
Oct 30 22:27:04.671149 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:04.671110    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d7cdcc-c769d69c6-7z9lp" event={"ID":"1686023a-bf9e-4224-82ea-1c72bc4273c6","Type":"ContainerStarted","Data":"45663b3cc455a133e35849a4c7acf727a79c51f365416d0d2bec750b0c18871b"}
Oct 30 22:27:04.671353 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:04.671335    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d7cdcc-c769d69c6-7z9lp"
Oct 30 22:27:06.349455 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:06.349366    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d7cdcc-c769d69c6-7z9lp" podStartSLOduration=4.44286502 podStartE2EDuration="5.349351437s" totalImagesPullingTime="906.486417ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:27:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:27:04.805687617 +0000 UTC m=+2202.409995171" watchObservedRunningTime="2026-10-30 22:27:06.349351437 +0000 UTC m=+2203.953658998"
Oct 30 22:27:06.350754 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:06.350728    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0b6a3f-f9776f9d7-rxn5c"]
Oct 30 22:27:06.354612 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:06.354591    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0b6a3f-f9776f9d7-rxn5c"
Oct 30 22:27:06.438417 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:06.438370    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df2fee05-175f-44c9-9640-b648c8744981-maas-api-tls\") pod \"maas-api-e2e-api-a-0b6a3f-f9776f9d7-rxn5c\" (UID: \"df2fee05-175f-44c9-9640-b648c8744981\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0b6a3f-f9776f9d7-rxn5c"
Oct 30 22:27:06.438595 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:06.438508    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hvvg\" (UniqueName: \"kubernetes.io/projected/df2fee05-175f-44c9-9640-b648c8744981-kube-api-access-8hvvg\") pod \"maas-api-e2e-api-a-0b6a3f-f9776f9d7-rxn5c\" (UID: \"df2fee05-175f-44c9-9640-b648c8744981\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0b6a3f-f9776f9d7-rxn5c"
Oct 30 22:27:06.438648 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:06.438631    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df2fee05-175f-44c9-9640-b648c8744981-metrics-certs\") pod \"maas-api-e2e-api-a-0b6a3f-f9776f9d7-rxn5c\" (UID: \"df2fee05-175f-44c9-9640-b648c8744981\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0b6a3f-f9776f9d7-rxn5c"
Oct 30 22:27:06.438801 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:06.438788    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df2fee05-175f-44c9-9640-b648c8744981-openshift-service-ca\") pod \"maas-api-e2e-api-a-0b6a3f-f9776f9d7-rxn5c\" (UID: \"df2fee05-175f-44c9-9640-b648c8744981\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0b6a3f-f9776f9d7-rxn5c"
Oct 30 22:27:06.454074 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:06.454015    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-0b6a3f-serving-cert\""
Oct 30 22:27:06.460431 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:06.460387    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0b6a3f-f9776f9d7-rxn5c"]
Oct 30 22:27:06.539672 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:06.539639    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df2fee05-175f-44c9-9640-b648c8744981-maas-api-tls\") pod \"maas-api-e2e-api-a-0b6a3f-f9776f9d7-rxn5c\" (UID: \"df2fee05-175f-44c9-9640-b648c8744981\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0b6a3f-f9776f9d7-rxn5c"
Oct 30 22:27:06.539873 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:06.539711    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hvvg\" (UniqueName: \"kubernetes.io/projected/df2fee05-175f-44c9-9640-b648c8744981-kube-api-access-8hvvg\") pod \"maas-api-e2e-api-a-0b6a3f-f9776f9d7-rxn5c\" (UID: \"df2fee05-175f-44c9-9640-b648c8744981\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0b6a3f-f9776f9d7-rxn5c"
Oct 30 22:27:06.539873 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:27:06.539772    5678 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-0b6a3f-serving-cert: secret "maas-api-e2e-api-a-0b6a3f-serving-cert" not found
Oct 30 22:27:06.539873 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:06.539775    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df2fee05-175f-44c9-9640-b648c8744981-metrics-certs\") pod \"maas-api-e2e-api-a-0b6a3f-f9776f9d7-rxn5c\" (UID: \"df2fee05-175f-44c9-9640-b648c8744981\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0b6a3f-f9776f9d7-rxn5c"
Oct 30 22:27:06.539873 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:27:06.539848    5678 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df2fee05-175f-44c9-9640-b648c8744981-maas-api-tls podName:df2fee05-175f-44c9-9640-b648c8744981 nodeName:}" failed. No retries permitted until 2026-10-30 22:27:07.039827419 +0000 UTC m=+2204.644134964 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/df2fee05-175f-44c9-9640-b648c8744981-maas-api-tls") pod "maas-api-e2e-api-a-0b6a3f-f9776f9d7-rxn5c" (UID: "df2fee05-175f-44c9-9640-b648c8744981") : secret "maas-api-e2e-api-a-0b6a3f-serving-cert" not found
Oct 30 22:27:06.540146 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:06.540020    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df2fee05-175f-44c9-9640-b648c8744981-openshift-service-ca\") pod \"maas-api-e2e-api-a-0b6a3f-f9776f9d7-rxn5c\" (UID: \"df2fee05-175f-44c9-9640-b648c8744981\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0b6a3f-f9776f9d7-rxn5c"
Oct 30 22:27:06.540705 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:06.540680    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df2fee05-175f-44c9-9640-b648c8744981-openshift-service-ca\") pod \"maas-api-e2e-api-a-0b6a3f-f9776f9d7-rxn5c\" (UID: \"df2fee05-175f-44c9-9640-b648c8744981\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0b6a3f-f9776f9d7-rxn5c"
Oct 30 22:27:06.542169 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:06.542149    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df2fee05-175f-44c9-9640-b648c8744981-metrics-certs\") pod \"maas-api-e2e-api-a-0b6a3f-f9776f9d7-rxn5c\" (UID: \"df2fee05-175f-44c9-9640-b648c8744981\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0b6a3f-f9776f9d7-rxn5c"
Oct 30 22:27:06.606176 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:06.606062    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hvvg\" (UniqueName: \"kubernetes.io/projected/df2fee05-175f-44c9-9640-b648c8744981-kube-api-access-8hvvg\") pod \"maas-api-e2e-api-a-0b6a3f-f9776f9d7-rxn5c\" (UID: \"df2fee05-175f-44c9-9640-b648c8744981\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0b6a3f-f9776f9d7-rxn5c"
Oct 30 22:27:07.049190 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:07.049147    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df2fee05-175f-44c9-9640-b648c8744981-maas-api-tls\") pod \"maas-api-e2e-api-a-0b6a3f-f9776f9d7-rxn5c\" (UID: \"df2fee05-175f-44c9-9640-b648c8744981\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0b6a3f-f9776f9d7-rxn5c"
Oct 30 22:27:07.051784 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:07.051750    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df2fee05-175f-44c9-9640-b648c8744981-maas-api-tls\") pod \"maas-api-e2e-api-a-0b6a3f-f9776f9d7-rxn5c\" (UID: \"df2fee05-175f-44c9-9640-b648c8744981\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0b6a3f-f9776f9d7-rxn5c"
Oct 30 22:27:07.267748 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:07.267705    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0b6a3f-f9776f9d7-rxn5c"
Oct 30 22:27:07.657259 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:07.657229    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0b6a3f-f9776f9d7-rxn5c"]
Oct 30 22:27:07.662830 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:27:07.662804    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf2fee05_175f_44c9_9640_b648c8744981.slice/crio-bdbee2d815452ec94b0411880d99d130d8158e89fa9da896042514f129547035 WatchSource:0}: Error finding container bdbee2d815452ec94b0411880d99d130d8158e89fa9da896042514f129547035: Status 404 returned error can't find the container with id bdbee2d815452ec94b0411880d99d130d8158e89fa9da896042514f129547035
Oct 30 22:27:07.692704 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:07.692671    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0b6a3f-f9776f9d7-rxn5c" event={"ID":"df2fee05-175f-44c9-9640-b648c8744981","Type":"ContainerStarted","Data":"bdbee2d815452ec94b0411880d99d130d8158e89fa9da896042514f129547035"}
Oct 30 22:27:08.702631 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:08.702588    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0b6a3f-f9776f9d7-rxn5c" event={"ID":"df2fee05-175f-44c9-9640-b648c8744981","Type":"ContainerStarted","Data":"30aff90aab0591f80371a2de554ba3559f08f4833a989bddbde0db1ffda1b062"}
Oct 30 22:27:08.703006 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:08.702796    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0b6a3f-f9776f9d7-rxn5c"
Oct 30 22:27:08.827855 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:08.827800    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0b6a3f-f9776f9d7-rxn5c" podStartSLOduration=1.893076754 podStartE2EDuration="2.827785s" totalImagesPullingTime="934.708246ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:27:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:27:08.82484824 +0000 UTC m=+2206.429155804" watchObservedRunningTime="2026-10-30 22:27:08.827785 +0000 UTC m=+2206.432092553"
Oct 30 22:27:09.898089 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:09.898058    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-2d6zs"]
Oct 30 22:27:09.898592 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:09.898266    5678 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-2d6zs" podUID="8061c232-3802-4de3-b53c-52238ba2bc75" containerName="istio-proxy" containerID="cri-o://04d94a695fde10287a9a81f3853795998ed2fc7f424bfdaff7a792c7d466a83e" gracePeriod=30
Oct 30 22:27:10.573327 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:27:10.573291    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 22:27:10.683590 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:10.683564    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d7cdcc-c769d69c6-7z9lp"
Oct 30 22:27:11.225464 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:11.225431    5678 patch_prober.go:29] interesting pod/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-2d6zs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:27:11.225854 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:11.225483    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-2d6zs" podUID="8061c232-3802-4de3-b53c-52238ba2bc75" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:27:14.715057 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:14.715028    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0b6a3f-f9776f9d7-rxn5c"
Oct 30 22:27:15.176093 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:15.176064    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-2d6zs"
Oct 30 22:27:15.283391 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:15.283308    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8061c232-3802-4de3-b53c-52238ba2bc75-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8061c232-3802-4de3-b53c-52238ba2bc75-istio-podinfo\") pod \"8061c232-3802-4de3-b53c-52238ba2bc75\" (UID: \"8061c232-3802-4de3-b53c-52238ba2bc75\") "
Oct 30 22:27:15.283391 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:15.283357    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8061c232-3802-4de3-b53c-52238ba2bc75-istio-token\" (UniqueName: \"kubernetes.io/projected/8061c232-3802-4de3-b53c-52238ba2bc75-istio-token\") pod \"8061c232-3802-4de3-b53c-52238ba2bc75\" (UID: \"8061c232-3802-4de3-b53c-52238ba2bc75\") "
Oct 30 22:27:15.283617 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:15.283504    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8061c232-3802-4de3-b53c-52238ba2bc75-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8061c232-3802-4de3-b53c-52238ba2bc75-workload-certs\") pod \"8061c232-3802-4de3-b53c-52238ba2bc75\" (UID: \"8061c232-3802-4de3-b53c-52238ba2bc75\") "
Oct 30 22:27:15.283617 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:15.283535    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8061c232-3802-4de3-b53c-52238ba2bc75-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8061c232-3802-4de3-b53c-52238ba2bc75-workload-socket\") pod \"8061c232-3802-4de3-b53c-52238ba2bc75\" (UID: \"8061c232-3802-4de3-b53c-52238ba2bc75\") "
Oct 30 22:27:15.283617 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:15.283578    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8061c232-3802-4de3-b53c-52238ba2bc75-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8061c232-3802-4de3-b53c-52238ba2bc75-istio-envoy\") pod \"8061c232-3802-4de3-b53c-52238ba2bc75\" (UID: \"8061c232-3802-4de3-b53c-52238ba2bc75\") "
Oct 30 22:27:15.283617 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:15.283607    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8061c232-3802-4de3-b53c-52238ba2bc75-kube-api-access-jb2pz\" (UniqueName: \"kubernetes.io/projected/8061c232-3802-4de3-b53c-52238ba2bc75-kube-api-access-jb2pz\") pod \"8061c232-3802-4de3-b53c-52238ba2bc75\" (UID: \"8061c232-3802-4de3-b53c-52238ba2bc75\") "
Oct 30 22:27:15.283824 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:15.283660    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8061c232-3802-4de3-b53c-52238ba2bc75-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8061c232-3802-4de3-b53c-52238ba2bc75-credential-socket\") pod \"8061c232-3802-4de3-b53c-52238ba2bc75\" (UID: \"8061c232-3802-4de3-b53c-52238ba2bc75\") "
Oct 30 22:27:15.283824 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:15.283763    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8061c232-3802-4de3-b53c-52238ba2bc75-workload-certs" pod "8061c232-3802-4de3-b53c-52238ba2bc75" (UID: "8061c232-3802-4de3-b53c-52238ba2bc75"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:27:15.283824 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:15.283801    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8061c232-3802-4de3-b53c-52238ba2bc75-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8061c232-3802-4de3-b53c-52238ba2bc75-istio-data\") pod \"8061c232-3802-4de3-b53c-52238ba2bc75\" (UID: \"8061c232-3802-4de3-b53c-52238ba2bc75\") "
Oct 30 22:27:15.283992 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:15.283868    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8061c232-3802-4de3-b53c-52238ba2bc75-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8061c232-3802-4de3-b53c-52238ba2bc75-istiod-ca-cert\") pod \"8061c232-3802-4de3-b53c-52238ba2bc75\" (UID: \"8061c232-3802-4de3-b53c-52238ba2bc75\") "
Oct 30 22:27:15.283992 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:15.283893    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8061c232-3802-4de3-b53c-52238ba2bc75-workload-socket" pod "8061c232-3802-4de3-b53c-52238ba2bc75" (UID: "8061c232-3802-4de3-b53c-52238ba2bc75"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:27:15.283992 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:15.283982    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8061c232-3802-4de3-b53c-52238ba2bc75-credential-socket" pod "8061c232-3802-4de3-b53c-52238ba2bc75" (UID: "8061c232-3802-4de3-b53c-52238ba2bc75"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:27:15.284165 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:15.284114    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8061c232-3802-4de3-b53c-52238ba2bc75-istio-data" pod "8061c232-3802-4de3-b53c-52238ba2bc75" (UID: "8061c232-3802-4de3-b53c-52238ba2bc75"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:27:15.284455 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:15.284434    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8061c232-3802-4de3-b53c-52238ba2bc75-istiod-ca-cert" pod "8061c232-3802-4de3-b53c-52238ba2bc75" (UID: "8061c232-3802-4de3-b53c-52238ba2bc75"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:27:15.284668 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:15.284644    5678 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8061c232-3802-4de3-b53c-52238ba2bc75-workload-certs\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:15.284737 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:15.284676    5678 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8061c232-3802-4de3-b53c-52238ba2bc75-workload-socket\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:15.284737 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:15.284690    5678 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8061c232-3802-4de3-b53c-52238ba2bc75-credential-socket\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:15.284737 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:15.284708    5678 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8061c232-3802-4de3-b53c-52238ba2bc75-istio-data\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:15.284737 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:15.284721    5678 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8061c232-3802-4de3-b53c-52238ba2bc75-istiod-ca-cert\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:15.286016 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:15.285961    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8061c232-3802-4de3-b53c-52238ba2bc75-istio-envoy" pod "8061c232-3802-4de3-b53c-52238ba2bc75" (UID: "8061c232-3802-4de3-b53c-52238ba2bc75"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:27:15.286125 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:15.286081    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8061c232-3802-4de3-b53c-52238ba2bc75-istio-token" pod "8061c232-3802-4de3-b53c-52238ba2bc75" (UID: "8061c232-3802-4de3-b53c-52238ba2bc75"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:27:15.286226 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:15.286206    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8061c232-3802-4de3-b53c-52238ba2bc75-kube-api-access-jb2pz" pod "8061c232-3802-4de3-b53c-52238ba2bc75" (UID: "8061c232-3802-4de3-b53c-52238ba2bc75"). InnerVolumeSpecName "kube-api-access-jb2pz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:27:15.286414 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:15.286383    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8061c232-3802-4de3-b53c-52238ba2bc75-istio-podinfo" pod "8061c232-3802-4de3-b53c-52238ba2bc75" (UID: "8061c232-3802-4de3-b53c-52238ba2bc75"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:27:15.386185 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:15.386150    5678 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8061c232-3802-4de3-b53c-52238ba2bc75-istio-envoy\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:15.386185 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:15.386181    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jb2pz\" (UniqueName: \"kubernetes.io/projected/8061c232-3802-4de3-b53c-52238ba2bc75-kube-api-access-jb2pz\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:15.386185 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:15.386192    5678 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8061c232-3802-4de3-b53c-52238ba2bc75-istio-podinfo\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:15.386435 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:15.386202    5678 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8061c232-3802-4de3-b53c-52238ba2bc75-istio-token\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:15.756906 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:15.756873    5678 generic.go:356] "Generic (PLEG): container finished" podID="8061c232-3802-4de3-b53c-52238ba2bc75" containerID="04d94a695fde10287a9a81f3853795998ed2fc7f424bfdaff7a792c7d466a83e" exitCode=0
Oct 30 22:27:15.757375 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:15.756929    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-2d6zs"
Oct 30 22:27:15.757375 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:15.756953    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-2d6zs" event={"ID":"8061c232-3802-4de3-b53c-52238ba2bc75","Type":"ContainerDied","Data":"04d94a695fde10287a9a81f3853795998ed2fc7f424bfdaff7a792c7d466a83e"}
Oct 30 22:27:15.757375 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:15.757024    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-2d6zs" event={"ID":"8061c232-3802-4de3-b53c-52238ba2bc75","Type":"ContainerDied","Data":"c7e88b11fd84835dc82e05d8cc522446cca366eac2c66bb55e34dd8b70e3d91d"}
Oct 30 22:27:15.757375 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:15.757056    5678 scope.go:122] "RemoveContainer" containerID="04d94a695fde10287a9a81f3853795998ed2fc7f424bfdaff7a792c7d466a83e"
Oct 30 22:27:15.772876 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:15.772857    5678 scope.go:122] "RemoveContainer" containerID="04d94a695fde10287a9a81f3853795998ed2fc7f424bfdaff7a792c7d466a83e"
Oct 30 22:27:15.773196 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:27:15.773176    5678 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04d94a695fde10287a9a81f3853795998ed2fc7f424bfdaff7a792c7d466a83e\": container with ID starting with 04d94a695fde10287a9a81f3853795998ed2fc7f424bfdaff7a792c7d466a83e not found: ID does not exist" containerID="04d94a695fde10287a9a81f3853795998ed2fc7f424bfdaff7a792c7d466a83e"
Oct 30 22:27:15.773250 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:15.773207    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04d94a695fde10287a9a81f3853795998ed2fc7f424bfdaff7a792c7d466a83e"} err="failed to get container status \"04d94a695fde10287a9a81f3853795998ed2fc7f424bfdaff7a792c7d466a83e\": rpc error: code = NotFound desc = could not find container \"04d94a695fde10287a9a81f3853795998ed2fc7f424bfdaff7a792c7d466a83e\": container with ID starting with 04d94a695fde10287a9a81f3853795998ed2fc7f424bfdaff7a792c7d466a83e not found: ID does not exist"
Oct 30 22:27:15.919702 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:15.919669    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-2d6zs"]
Oct 30 22:27:15.963802 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:15.963768    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-2d6zs"]
Oct 30 22:27:16.524441 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:16.524412    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8061c232-3802-4de3-b53c-52238ba2bc75" path="/var/lib/kubelet/pods/8061c232-3802-4de3-b53c-52238ba2bc75/volumes"
Oct 30 22:27:23.274149 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:23.274115    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bwbzj"]
Oct 30 22:27:23.274726 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:23.274323    5678 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bwbzj" podUID="62c63093-87f0-4e2d-81b7-5f8cc31cee82" containerName="istio-proxy" containerID="cri-o://047cf19b07e620417137aa117f6984b58ea0d7be0ad9e06f5bb6a795d04fc3c5" gracePeriod=30
Oct 30 22:27:23.695430 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:23.695373    5678 scope.go:122] "RemoveContainer" containerID="2a09d6934cb79833e0a16eba32df437c3f53da4db96843f212129ad58aa81a7b"
Oct 30 22:27:25.198666 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:25.198631    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-f2qtx"]
Oct 30 22:27:25.199158 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:25.198884    5678 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-f2qtx" podUID="95f62fb3-9a67-4bca-9483-8aedadf82cd6" containerName="istio-proxy" containerID="cri-o://6a9c06c3d44802378cdacb2dea0eb91d1a51c076fa1781a621dbffeb2e301866" gracePeriod=30
Oct 30 22:27:26.077709 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:26.077673    5678 patch_prober.go:29] interesting pod/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-f2qtx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:27:26.077881 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:26.077724    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-f2qtx" podUID="95f62fb3-9a67-4bca-9483-8aedadf82cd6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:27:26.287320 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:26.287282    5678 patch_prober.go:29] interesting pod/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bwbzj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:27:26.287732 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:26.287362    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bwbzj" podUID="62c63093-87f0-4e2d-81b7-5f8cc31cee82" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:27:28.552085 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:28.552023    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bwbzj"
Oct 30 22:27:28.584205 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:28.584175    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62c63093-87f0-4e2d-81b7-5f8cc31cee82-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62c63093-87f0-4e2d-81b7-5f8cc31cee82-credential-socket\") pod \"62c63093-87f0-4e2d-81b7-5f8cc31cee82\" (UID: \"62c63093-87f0-4e2d-81b7-5f8cc31cee82\") "
Oct 30 22:27:28.584396 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:28.584214    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62c63093-87f0-4e2d-81b7-5f8cc31cee82-istio-token\" (UniqueName: \"kubernetes.io/projected/62c63093-87f0-4e2d-81b7-5f8cc31cee82-istio-token\") pod \"62c63093-87f0-4e2d-81b7-5f8cc31cee82\" (UID: \"62c63093-87f0-4e2d-81b7-5f8cc31cee82\") "
Oct 30 22:27:28.584396 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:28.584244    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62c63093-87f0-4e2d-81b7-5f8cc31cee82-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62c63093-87f0-4e2d-81b7-5f8cc31cee82-workload-certs\") pod \"62c63093-87f0-4e2d-81b7-5f8cc31cee82\" (UID: \"62c63093-87f0-4e2d-81b7-5f8cc31cee82\") "
Oct 30 22:27:28.584396 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:28.584265    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62c63093-87f0-4e2d-81b7-5f8cc31cee82-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62c63093-87f0-4e2d-81b7-5f8cc31cee82-workload-socket\") pod \"62c63093-87f0-4e2d-81b7-5f8cc31cee82\" (UID: \"62c63093-87f0-4e2d-81b7-5f8cc31cee82\") "
Oct 30 22:27:28.584396 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:28.584284    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62c63093-87f0-4e2d-81b7-5f8cc31cee82-kube-api-access-7jhr9\" (UniqueName: \"kubernetes.io/projected/62c63093-87f0-4e2d-81b7-5f8cc31cee82-kube-api-access-7jhr9\") pod \"62c63093-87f0-4e2d-81b7-5f8cc31cee82\" (UID: \"62c63093-87f0-4e2d-81b7-5f8cc31cee82\") "
Oct 30 22:27:28.584396 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:28.584383    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62c63093-87f0-4e2d-81b7-5f8cc31cee82-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62c63093-87f0-4e2d-81b7-5f8cc31cee82-istio-envoy\") pod \"62c63093-87f0-4e2d-81b7-5f8cc31cee82\" (UID: \"62c63093-87f0-4e2d-81b7-5f8cc31cee82\") "
Oct 30 22:27:28.584703 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:28.584493    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62c63093-87f0-4e2d-81b7-5f8cc31cee82-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62c63093-87f0-4e2d-81b7-5f8cc31cee82-istio-data\") pod \"62c63093-87f0-4e2d-81b7-5f8cc31cee82\" (UID: \"62c63093-87f0-4e2d-81b7-5f8cc31cee82\") "
Oct 30 22:27:28.584703 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:28.584631    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62c63093-87f0-4e2d-81b7-5f8cc31cee82-workload-certs" pod "62c63093-87f0-4e2d-81b7-5f8cc31cee82" (UID: "62c63093-87f0-4e2d-81b7-5f8cc31cee82"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:27:28.584703 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:28.584640    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62c63093-87f0-4e2d-81b7-5f8cc31cee82-workload-socket" pod "62c63093-87f0-4e2d-81b7-5f8cc31cee82" (UID: "62c63093-87f0-4e2d-81b7-5f8cc31cee82"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:27:28.584703 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:28.584661    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/62c63093-87f0-4e2d-81b7-5f8cc31cee82-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62c63093-87f0-4e2d-81b7-5f8cc31cee82-istio-podinfo\") pod \"62c63093-87f0-4e2d-81b7-5f8cc31cee82\" (UID: \"62c63093-87f0-4e2d-81b7-5f8cc31cee82\") "
Oct 30 22:27:28.584703 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:28.584702    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62c63093-87f0-4e2d-81b7-5f8cc31cee82-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62c63093-87f0-4e2d-81b7-5f8cc31cee82-istiod-ca-cert\") pod \"62c63093-87f0-4e2d-81b7-5f8cc31cee82\" (UID: \"62c63093-87f0-4e2d-81b7-5f8cc31cee82\") "
Oct 30 22:27:28.584971 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:28.584846    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62c63093-87f0-4e2d-81b7-5f8cc31cee82-istio-data" pod "62c63093-87f0-4e2d-81b7-5f8cc31cee82" (UID: "62c63093-87f0-4e2d-81b7-5f8cc31cee82"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:27:28.584971 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:28.584888    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62c63093-87f0-4e2d-81b7-5f8cc31cee82-credential-socket" pod "62c63093-87f0-4e2d-81b7-5f8cc31cee82" (UID: "62c63093-87f0-4e2d-81b7-5f8cc31cee82"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:27:28.585219 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:28.585195    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62c63093-87f0-4e2d-81b7-5f8cc31cee82-istiod-ca-cert" pod "62c63093-87f0-4e2d-81b7-5f8cc31cee82" (UID: "62c63093-87f0-4e2d-81b7-5f8cc31cee82"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:27:28.585692 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:28.585670    5678 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62c63093-87f0-4e2d-81b7-5f8cc31cee82-istiod-ca-cert\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:28.585820 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:28.585696    5678 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62c63093-87f0-4e2d-81b7-5f8cc31cee82-credential-socket\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:28.585820 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:28.585710    5678 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62c63093-87f0-4e2d-81b7-5f8cc31cee82-workload-certs\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:28.585820 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:28.585723    5678 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62c63093-87f0-4e2d-81b7-5f8cc31cee82-workload-socket\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:28.585820 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:28.585737    5678 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62c63093-87f0-4e2d-81b7-5f8cc31cee82-istio-data\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:28.586828 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:28.586802    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/62c63093-87f0-4e2d-81b7-5f8cc31cee82-istio-podinfo" pod "62c63093-87f0-4e2d-81b7-5f8cc31cee82" (UID: "62c63093-87f0-4e2d-81b7-5f8cc31cee82"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:27:28.587006 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:28.586987    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62c63093-87f0-4e2d-81b7-5f8cc31cee82-kube-api-access-7jhr9" pod "62c63093-87f0-4e2d-81b7-5f8cc31cee82" (UID: "62c63093-87f0-4e2d-81b7-5f8cc31cee82"). InnerVolumeSpecName "kube-api-access-7jhr9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:27:28.587088 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:28.587029    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62c63093-87f0-4e2d-81b7-5f8cc31cee82-istio-token" pod "62c63093-87f0-4e2d-81b7-5f8cc31cee82" (UID: "62c63093-87f0-4e2d-81b7-5f8cc31cee82"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:27:28.587088 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:28.587056    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62c63093-87f0-4e2d-81b7-5f8cc31cee82-istio-envoy" pod "62c63093-87f0-4e2d-81b7-5f8cc31cee82" (UID: "62c63093-87f0-4e2d-81b7-5f8cc31cee82"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:27:28.686678 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:28.686646    5678 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62c63093-87f0-4e2d-81b7-5f8cc31cee82-istio-podinfo\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:28.686678 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:28.686671    5678 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62c63093-87f0-4e2d-81b7-5f8cc31cee82-istio-token\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:28.686678 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:28.686681    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jhr9\" (UniqueName: \"kubernetes.io/projected/62c63093-87f0-4e2d-81b7-5f8cc31cee82-kube-api-access-7jhr9\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:28.686900 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:28.686692    5678 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62c63093-87f0-4e2d-81b7-5f8cc31cee82-istio-envoy\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:28.854283 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:28.854199    5678 generic.go:356] "Generic (PLEG): container finished" podID="62c63093-87f0-4e2d-81b7-5f8cc31cee82" containerID="047cf19b07e620417137aa117f6984b58ea0d7be0ad9e06f5bb6a795d04fc3c5" exitCode=0
Oct 30 22:27:28.854283 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:28.854246    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bwbzj"
Oct 30 22:27:28.854525 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:28.854281    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bwbzj" event={"ID":"62c63093-87f0-4e2d-81b7-5f8cc31cee82","Type":"ContainerDied","Data":"047cf19b07e620417137aa117f6984b58ea0d7be0ad9e06f5bb6a795d04fc3c5"}
Oct 30 22:27:28.854525 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:28.854335    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bwbzj" event={"ID":"62c63093-87f0-4e2d-81b7-5f8cc31cee82","Type":"ContainerDied","Data":"fd0e5ceb882a5175b2998655952f73bef91f8b610e071ce09bb0a4b8ae686805"}
Oct 30 22:27:28.854525 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:28.854357    5678 scope.go:122] "RemoveContainer" containerID="047cf19b07e620417137aa117f6984b58ea0d7be0ad9e06f5bb6a795d04fc3c5"
Oct 30 22:27:28.865630 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:28.865613    5678 scope.go:122] "RemoveContainer" containerID="047cf19b07e620417137aa117f6984b58ea0d7be0ad9e06f5bb6a795d04fc3c5"
Oct 30 22:27:28.865908 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:27:28.865892    5678 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"047cf19b07e620417137aa117f6984b58ea0d7be0ad9e06f5bb6a795d04fc3c5\": container with ID starting with 047cf19b07e620417137aa117f6984b58ea0d7be0ad9e06f5bb6a795d04fc3c5 not found: ID does not exist" containerID="047cf19b07e620417137aa117f6984b58ea0d7be0ad9e06f5bb6a795d04fc3c5"
Oct 30 22:27:28.865957 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:28.865916    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"047cf19b07e620417137aa117f6984b58ea0d7be0ad9e06f5bb6a795d04fc3c5"} err="failed to get container status \"047cf19b07e620417137aa117f6984b58ea0d7be0ad9e06f5bb6a795d04fc3c5\": rpc error: code = NotFound desc = could not find container \"047cf19b07e620417137aa117f6984b58ea0d7be0ad9e06f5bb6a795d04fc3c5\": container with ID starting with 047cf19b07e620417137aa117f6984b58ea0d7be0ad9e06f5bb6a795d04fc3c5 not found: ID does not exist"
Oct 30 22:27:29.037254 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:29.037221    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bwbzj"]
Oct 30 22:27:29.042626 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:29.042602    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bwbzj"]
Oct 30 22:27:30.470052 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:30.470027    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-f2qtx"
Oct 30 22:27:30.525157 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:30.525116    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62c63093-87f0-4e2d-81b7-5f8cc31cee82" path="/var/lib/kubelet/pods/62c63093-87f0-4e2d-81b7-5f8cc31cee82/volumes"
Oct 30 22:27:30.532747 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:30.532715    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95f62fb3-9a67-4bca-9483-8aedadf82cd6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95f62fb3-9a67-4bca-9483-8aedadf82cd6-istio-envoy\") pod \"95f62fb3-9a67-4bca-9483-8aedadf82cd6\" (UID: \"95f62fb3-9a67-4bca-9483-8aedadf82cd6\") "
Oct 30 22:27:30.532882 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:30.532776    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95f62fb3-9a67-4bca-9483-8aedadf82cd6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95f62fb3-9a67-4bca-9483-8aedadf82cd6-workload-socket\") pod \"95f62fb3-9a67-4bca-9483-8aedadf82cd6\" (UID: \"95f62fb3-9a67-4bca-9483-8aedadf82cd6\") "
Oct 30 22:27:30.532937 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:30.532884    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95f62fb3-9a67-4bca-9483-8aedadf82cd6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95f62fb3-9a67-4bca-9483-8aedadf82cd6-credential-socket\") pod \"95f62fb3-9a67-4bca-9483-8aedadf82cd6\" (UID: \"95f62fb3-9a67-4bca-9483-8aedadf82cd6\") "
Oct 30 22:27:30.532993 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:30.532939    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95f62fb3-9a67-4bca-9483-8aedadf82cd6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95f62fb3-9a67-4bca-9483-8aedadf82cd6-workload-certs\") pod \"95f62fb3-9a67-4bca-9483-8aedadf82cd6\" (UID: \"95f62fb3-9a67-4bca-9483-8aedadf82cd6\") "
Oct 30 22:27:30.533048 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:30.533006    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95f62fb3-9a67-4bca-9483-8aedadf82cd6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95f62fb3-9a67-4bca-9483-8aedadf82cd6-istiod-ca-cert\") pod \"95f62fb3-9a67-4bca-9483-8aedadf82cd6\" (UID: \"95f62fb3-9a67-4bca-9483-8aedadf82cd6\") "
Oct 30 22:27:30.533048 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:30.533041    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95f62fb3-9a67-4bca-9483-8aedadf82cd6-istio-token\" (UniqueName: \"kubernetes.io/projected/95f62fb3-9a67-4bca-9483-8aedadf82cd6-istio-token\") pod \"95f62fb3-9a67-4bca-9483-8aedadf82cd6\" (UID: \"95f62fb3-9a67-4bca-9483-8aedadf82cd6\") "
Oct 30 22:27:30.533157 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:30.533101    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95f62fb3-9a67-4bca-9483-8aedadf82cd6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95f62fb3-9a67-4bca-9483-8aedadf82cd6-istio-data\") pod \"95f62fb3-9a67-4bca-9483-8aedadf82cd6\" (UID: \"95f62fb3-9a67-4bca-9483-8aedadf82cd6\") "
Oct 30 22:27:30.533157 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:30.533131    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/95f62fb3-9a67-4bca-9483-8aedadf82cd6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95f62fb3-9a67-4bca-9483-8aedadf82cd6-istio-podinfo\") pod \"95f62fb3-9a67-4bca-9483-8aedadf82cd6\" (UID: \"95f62fb3-9a67-4bca-9483-8aedadf82cd6\") "
Oct 30 22:27:30.533286 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:30.533219    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95f62fb3-9a67-4bca-9483-8aedadf82cd6-credential-socket" pod "95f62fb3-9a67-4bca-9483-8aedadf82cd6" (UID: "95f62fb3-9a67-4bca-9483-8aedadf82cd6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:27:30.533347 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:30.533301    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95f62fb3-9a67-4bca-9483-8aedadf82cd6-kube-api-access-w5fgs\" (UniqueName: \"kubernetes.io/projected/95f62fb3-9a67-4bca-9483-8aedadf82cd6-kube-api-access-w5fgs\") pod \"95f62fb3-9a67-4bca-9483-8aedadf82cd6\" (UID: \"95f62fb3-9a67-4bca-9483-8aedadf82cd6\") "
Oct 30 22:27:30.533434 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:30.533391    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95f62fb3-9a67-4bca-9483-8aedadf82cd6-istiod-ca-cert" pod "95f62fb3-9a67-4bca-9483-8aedadf82cd6" (UID: "95f62fb3-9a67-4bca-9483-8aedadf82cd6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:27:30.533536 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:30.533514    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95f62fb3-9a67-4bca-9483-8aedadf82cd6-workload-certs" pod "95f62fb3-9a67-4bca-9483-8aedadf82cd6" (UID: "95f62fb3-9a67-4bca-9483-8aedadf82cd6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:27:30.533536 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:30.533526    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95f62fb3-9a67-4bca-9483-8aedadf82cd6-istio-data" pod "95f62fb3-9a67-4bca-9483-8aedadf82cd6" (UID: "95f62fb3-9a67-4bca-9483-8aedadf82cd6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:27:30.534135 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:30.533832    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95f62fb3-9a67-4bca-9483-8aedadf82cd6-workload-socket" pod "95f62fb3-9a67-4bca-9483-8aedadf82cd6" (UID: "95f62fb3-9a67-4bca-9483-8aedadf82cd6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:27:30.534135 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:30.534013    5678 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95f62fb3-9a67-4bca-9483-8aedadf82cd6-workload-certs\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:30.534135 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:30.534037    5678 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95f62fb3-9a67-4bca-9483-8aedadf82cd6-istiod-ca-cert\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:30.534135 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:30.534052    5678 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95f62fb3-9a67-4bca-9483-8aedadf82cd6-istio-data\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:30.534135 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:30.534065    5678 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95f62fb3-9a67-4bca-9483-8aedadf82cd6-workload-socket\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:30.534135 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:30.534079    5678 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95f62fb3-9a67-4bca-9483-8aedadf82cd6-credential-socket\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:30.536182 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:30.536127    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95f62fb3-9a67-4bca-9483-8aedadf82cd6-istio-envoy" pod "95f62fb3-9a67-4bca-9483-8aedadf82cd6" (UID: "95f62fb3-9a67-4bca-9483-8aedadf82cd6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:27:30.536182 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:30.536162    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95f62fb3-9a67-4bca-9483-8aedadf82cd6-kube-api-access-w5fgs" pod "95f62fb3-9a67-4bca-9483-8aedadf82cd6" (UID: "95f62fb3-9a67-4bca-9483-8aedadf82cd6"). InnerVolumeSpecName "kube-api-access-w5fgs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:27:30.536348 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:30.536181    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/95f62fb3-9a67-4bca-9483-8aedadf82cd6-istio-podinfo" pod "95f62fb3-9a67-4bca-9483-8aedadf82cd6" (UID: "95f62fb3-9a67-4bca-9483-8aedadf82cd6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:27:30.536682 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:30.536665    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95f62fb3-9a67-4bca-9483-8aedadf82cd6-istio-token" pod "95f62fb3-9a67-4bca-9483-8aedadf82cd6" (UID: "95f62fb3-9a67-4bca-9483-8aedadf82cd6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:27:30.635570 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:30.635536    5678 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95f62fb3-9a67-4bca-9483-8aedadf82cd6-istio-token\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:30.635570 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:30.635569    5678 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95f62fb3-9a67-4bca-9483-8aedadf82cd6-istio-podinfo\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:30.635806 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:30.635583    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5fgs\" (UniqueName: \"kubernetes.io/projected/95f62fb3-9a67-4bca-9483-8aedadf82cd6-kube-api-access-w5fgs\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:30.635806 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:30.635598    5678 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95f62fb3-9a67-4bca-9483-8aedadf82cd6-istio-envoy\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:30.873179 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:30.873141    5678 generic.go:356] "Generic (PLEG): container finished" podID="95f62fb3-9a67-4bca-9483-8aedadf82cd6" containerID="6a9c06c3d44802378cdacb2dea0eb91d1a51c076fa1781a621dbffeb2e301866" exitCode=0
Oct 30 22:27:30.874032 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:30.873204    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-f2qtx"
Oct 30 22:27:30.874032 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:30.873236    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-f2qtx" event={"ID":"95f62fb3-9a67-4bca-9483-8aedadf82cd6","Type":"ContainerDied","Data":"6a9c06c3d44802378cdacb2dea0eb91d1a51c076fa1781a621dbffeb2e301866"}
Oct 30 22:27:30.874032 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:30.873298    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-f2qtx" event={"ID":"95f62fb3-9a67-4bca-9483-8aedadf82cd6","Type":"ContainerDied","Data":"c230e28df08b368fcbdebf557e259c7b9ca4e8bba59d2923d9395550b96560c9"}
Oct 30 22:27:30.874032 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:30.873329    5678 scope.go:122] "RemoveContainer" containerID="6a9c06c3d44802378cdacb2dea0eb91d1a51c076fa1781a621dbffeb2e301866"
Oct 30 22:27:30.885278 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:30.885261    5678 scope.go:122] "RemoveContainer" containerID="6a9c06c3d44802378cdacb2dea0eb91d1a51c076fa1781a621dbffeb2e301866"
Oct 30 22:27:30.885639 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:27:30.885618    5678 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a9c06c3d44802378cdacb2dea0eb91d1a51c076fa1781a621dbffeb2e301866\": container with ID starting with 6a9c06c3d44802378cdacb2dea0eb91d1a51c076fa1781a621dbffeb2e301866 not found: ID does not exist" containerID="6a9c06c3d44802378cdacb2dea0eb91d1a51c076fa1781a621dbffeb2e301866"
Oct 30 22:27:30.885715 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:30.885644    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a9c06c3d44802378cdacb2dea0eb91d1a51c076fa1781a621dbffeb2e301866"} err="failed to get container status \"6a9c06c3d44802378cdacb2dea0eb91d1a51c076fa1781a621dbffeb2e301866\": rpc error: code = NotFound desc = could not find container \"6a9c06c3d44802378cdacb2dea0eb91d1a51c076fa1781a621dbffeb2e301866\": container with ID starting with 6a9c06c3d44802378cdacb2dea0eb91d1a51c076fa1781a621dbffeb2e301866 not found: ID does not exist"
Oct 30 22:27:31.034686 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:31.034655    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-f2qtx"]
Oct 30 22:27:31.041632 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:31.041608    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-f2qtx"]
Oct 30 22:27:32.526141 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:32.526108    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95f62fb3-9a67-4bca-9483-8aedadf82cd6" path="/var/lib/kubelet/pods/95f62fb3-9a67-4bca-9483-8aedadf82cd6/volumes"
Oct 30 22:27:40.572673 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:27:40.572636    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 22:27:40.772446 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:40.772383    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-5637a7-openshift-default-5b47575485-dxr9g"]
Oct 30 22:27:40.772654 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:40.772630    5678 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-5637a7-openshift-default-5b47575485-dxr9g" podUID="9c5d0357-a8f0-484c-9584-4b0184fd5284" containerName="istio-proxy" containerID="cri-o://8a0e4e964afec05f0364932bdf0add52bb8c1557985001aa137064730590af54" gracePeriod=30
Oct 30 22:27:40.776108 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:40.776080    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-595fd7d7-656b887dc9-5fgv6"]
Oct 30 22:27:40.776825 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:40.776807    5678 cpu_manager.go:414] "containerMap: removing container" podUID="8061c232-3802-4de3-b53c-52238ba2bc75" containerName="istio-proxy"
Oct 30 22:27:40.776879 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:40.776826    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8061c232-3802-4de3-b53c-52238ba2bc75" containerName="istio-proxy"
Oct 30 22:27:40.776879 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:40.776845    5678 cpu_manager.go:414] "containerMap: removing container" podUID="95f62fb3-9a67-4bca-9483-8aedadf82cd6" containerName="istio-proxy"
Oct 30 22:27:40.776879 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:40.776850    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95f62fb3-9a67-4bca-9483-8aedadf82cd6" containerName="istio-proxy"
Oct 30 22:27:40.776879 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:40.776859    5678 cpu_manager.go:414] "containerMap: removing container" podUID="62c63093-87f0-4e2d-81b7-5f8cc31cee82" containerName="istio-proxy"
Oct 30 22:27:40.776879 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:40.776864    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62c63093-87f0-4e2d-81b7-5f8cc31cee82" containerName="istio-proxy"
Oct 30 22:27:40.777027 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:40.776978    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="8061c232-3802-4de3-b53c-52238ba2bc75" containerName="istio-proxy"
Oct 30 22:27:40.777027 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:40.776989    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="95f62fb3-9a67-4bca-9483-8aedadf82cd6" containerName="istio-proxy"
Oct 30 22:27:40.777027 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:40.776998    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="62c63093-87f0-4e2d-81b7-5f8cc31cee82" containerName="istio-proxy"
Oct 30 22:27:40.780845 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:40.780827    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-595fd7d7-656b887dc9-5fgv6"
Oct 30 22:27:40.798197 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:40.798173    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c5d99a1-0c2f-4e44-87d0-3da5c6ee4a6d-maas-api-tls\") pod \"maas-api-e2e-mt-595fd7d7-656b887dc9-5fgv6\" (UID: \"7c5d99a1-0c2f-4e44-87d0-3da5c6ee4a6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-595fd7d7-656b887dc9-5fgv6"
Oct 30 22:27:40.798308 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:40.798225    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c5d99a1-0c2f-4e44-87d0-3da5c6ee4a6d-metrics-certs\") pod \"maas-api-e2e-mt-595fd7d7-656b887dc9-5fgv6\" (UID: \"7c5d99a1-0c2f-4e44-87d0-3da5c6ee4a6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-595fd7d7-656b887dc9-5fgv6"
Oct 30 22:27:40.798308 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:40.798246    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c5d99a1-0c2f-4e44-87d0-3da5c6ee4a6d-openshift-service-ca\") pod \"maas-api-e2e-mt-595fd7d7-656b887dc9-5fgv6\" (UID: \"7c5d99a1-0c2f-4e44-87d0-3da5c6ee4a6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-595fd7d7-656b887dc9-5fgv6"
Oct 30 22:27:40.798442 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:40.798396    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7khcg\" (UniqueName: \"kubernetes.io/projected/7c5d99a1-0c2f-4e44-87d0-3da5c6ee4a6d-kube-api-access-7khcg\") pod \"maas-api-e2e-mt-595fd7d7-656b887dc9-5fgv6\" (UID: \"7c5d99a1-0c2f-4e44-87d0-3da5c6ee4a6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-595fd7d7-656b887dc9-5fgv6"
Oct 30 22:27:40.838181 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:40.838048    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-595fd7d7-serving-cert\""
Oct 30 22:27:40.903574 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:40.900391    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c5d99a1-0c2f-4e44-87d0-3da5c6ee4a6d-maas-api-tls\") pod \"maas-api-e2e-mt-595fd7d7-656b887dc9-5fgv6\" (UID: \"7c5d99a1-0c2f-4e44-87d0-3da5c6ee4a6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-595fd7d7-656b887dc9-5fgv6"
Oct 30 22:27:40.903574 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:40.900488    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c5d99a1-0c2f-4e44-87d0-3da5c6ee4a6d-metrics-certs\") pod \"maas-api-e2e-mt-595fd7d7-656b887dc9-5fgv6\" (UID: \"7c5d99a1-0c2f-4e44-87d0-3da5c6ee4a6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-595fd7d7-656b887dc9-5fgv6"
Oct 30 22:27:40.903574 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:40.900518    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c5d99a1-0c2f-4e44-87d0-3da5c6ee4a6d-openshift-service-ca\") pod \"maas-api-e2e-mt-595fd7d7-656b887dc9-5fgv6\" (UID: \"7c5d99a1-0c2f-4e44-87d0-3da5c6ee4a6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-595fd7d7-656b887dc9-5fgv6"
Oct 30 22:27:40.903574 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:40.900590    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7khcg\" (UniqueName: \"kubernetes.io/projected/7c5d99a1-0c2f-4e44-87d0-3da5c6ee4a6d-kube-api-access-7khcg\") pod \"maas-api-e2e-mt-595fd7d7-656b887dc9-5fgv6\" (UID: \"7c5d99a1-0c2f-4e44-87d0-3da5c6ee4a6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-595fd7d7-656b887dc9-5fgv6"
Oct 30 22:27:40.903574 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:40.901728    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c5d99a1-0c2f-4e44-87d0-3da5c6ee4a6d-openshift-service-ca\") pod \"maas-api-e2e-mt-595fd7d7-656b887dc9-5fgv6\" (UID: \"7c5d99a1-0c2f-4e44-87d0-3da5c6ee4a6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-595fd7d7-656b887dc9-5fgv6"
Oct 30 22:27:40.903574 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:27:40.901839    5678 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-595fd7d7-serving-cert: secret "maas-api-e2e-mt-595fd7d7-serving-cert" not found
Oct 30 22:27:40.903574 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:27:40.901894    5678 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c5d99a1-0c2f-4e44-87d0-3da5c6ee4a6d-maas-api-tls podName:7c5d99a1-0c2f-4e44-87d0-3da5c6ee4a6d nodeName:}" failed. No retries permitted until 2026-10-30 22:27:41.401875509 +0000 UTC m=+2239.006183044 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7c5d99a1-0c2f-4e44-87d0-3da5c6ee4a6d-maas-api-tls") pod "maas-api-e2e-mt-595fd7d7-656b887dc9-5fgv6" (UID: "7c5d99a1-0c2f-4e44-87d0-3da5c6ee4a6d") : secret "maas-api-e2e-mt-595fd7d7-serving-cert" not found
Oct 30 22:27:40.949464 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:40.915109    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-595fd7d7-656b887dc9-5fgv6"]
Oct 30 22:27:40.994430 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:40.993036    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c5d99a1-0c2f-4e44-87d0-3da5c6ee4a6d-metrics-certs\") pod \"maas-api-e2e-mt-595fd7d7-656b887dc9-5fgv6\" (UID: \"7c5d99a1-0c2f-4e44-87d0-3da5c6ee4a6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-595fd7d7-656b887dc9-5fgv6"
Oct 30 22:27:41.020443 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:41.017958    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7khcg\" (UniqueName: \"kubernetes.io/projected/7c5d99a1-0c2f-4e44-87d0-3da5c6ee4a6d-kube-api-access-7khcg\") pod \"maas-api-e2e-mt-595fd7d7-656b887dc9-5fgv6\" (UID: \"7c5d99a1-0c2f-4e44-87d0-3da5c6ee4a6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-595fd7d7-656b887dc9-5fgv6"
Oct 30 22:27:41.428037 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:41.427992    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c5d99a1-0c2f-4e44-87d0-3da5c6ee4a6d-maas-api-tls\") pod \"maas-api-e2e-mt-595fd7d7-656b887dc9-5fgv6\" (UID: \"7c5d99a1-0c2f-4e44-87d0-3da5c6ee4a6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-595fd7d7-656b887dc9-5fgv6"
Oct 30 22:27:41.430411 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:41.430380    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c5d99a1-0c2f-4e44-87d0-3da5c6ee4a6d-maas-api-tls\") pod \"maas-api-e2e-mt-595fd7d7-656b887dc9-5fgv6\" (UID: \"7c5d99a1-0c2f-4e44-87d0-3da5c6ee4a6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-595fd7d7-656b887dc9-5fgv6"
Oct 30 22:27:41.694667 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:41.694580    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-595fd7d7-656b887dc9-5fgv6"
Oct 30 22:27:42.088630 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:42.088551    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-595fd7d7-656b887dc9-5fgv6"]
Oct 30 22:27:42.093225 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:27:42.093197    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c5d99a1_0c2f_4e44_87d0_3da5c6ee4a6d.slice/crio-c2df2c0f43b6682cadf073362da8ca44972f40c3f18b95e80522e15b817cfc80 WatchSource:0}: Error finding container c2df2c0f43b6682cadf073362da8ca44972f40c3f18b95e80522e15b817cfc80: Status 404 returned error can't find the container with id c2df2c0f43b6682cadf073362da8ca44972f40c3f18b95e80522e15b817cfc80
Oct 30 22:27:42.438038 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:42.437705    5678 patch_prober.go:29] interesting pod/e2e-inf-a-5637a7-openshift-default-5b47575485-dxr9g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:27:42.438038 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:42.437821    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-5637a7-openshift-default-5b47575485-dxr9g" podUID="9c5d0357-a8f0-484c-9584-4b0184fd5284" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:27:43.098471 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:43.098299    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-595fd7d7-656b887dc9-5fgv6" event={"ID":"7c5d99a1-0c2f-4e44-87d0-3da5c6ee4a6d","Type":"ContainerStarted","Data":"c2df2c0f43b6682cadf073362da8ca44972f40c3f18b95e80522e15b817cfc80"}
Oct 30 22:27:44.108396 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:44.108353    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-595fd7d7-656b887dc9-5fgv6" event={"ID":"7c5d99a1-0c2f-4e44-87d0-3da5c6ee4a6d","Type":"ContainerStarted","Data":"b23b131e739340e130c7c41ba801b565b831ae7101cc5e65cb1ea8791607b578"}
Oct 30 22:27:44.108808 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:44.108469    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-595fd7d7-656b887dc9-5fgv6"
Oct 30 22:27:46.060682 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:46.060658    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5637a7-openshift-default-5b47575485-dxr9g"
Oct 30 22:27:46.113593 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:46.113569    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c5d0357-a8f0-484c-9584-4b0184fd5284-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c5d0357-a8f0-484c-9584-4b0184fd5284-istiod-ca-cert\") pod \"9c5d0357-a8f0-484c-9584-4b0184fd5284\" (UID: \"9c5d0357-a8f0-484c-9584-4b0184fd5284\") "
Oct 30 22:27:46.113750 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:46.113604    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c5d0357-a8f0-484c-9584-4b0184fd5284-kube-api-access-s8kqp\" (UniqueName: \"kubernetes.io/projected/9c5d0357-a8f0-484c-9584-4b0184fd5284-kube-api-access-s8kqp\") pod \"9c5d0357-a8f0-484c-9584-4b0184fd5284\" (UID: \"9c5d0357-a8f0-484c-9584-4b0184fd5284\") "
Oct 30 22:27:46.113750 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:46.113700    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c5d0357-a8f0-484c-9584-4b0184fd5284-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c5d0357-a8f0-484c-9584-4b0184fd5284-workload-certs\") pod \"9c5d0357-a8f0-484c-9584-4b0184fd5284\" (UID: \"9c5d0357-a8f0-484c-9584-4b0184fd5284\") "
Oct 30 22:27:46.113750 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:46.113728    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c5d0357-a8f0-484c-9584-4b0184fd5284-istio-token\" (UniqueName: \"kubernetes.io/projected/9c5d0357-a8f0-484c-9584-4b0184fd5284-istio-token\") pod \"9c5d0357-a8f0-484c-9584-4b0184fd5284\" (UID: \"9c5d0357-a8f0-484c-9584-4b0184fd5284\") "
Oct 30 22:27:46.113750 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:46.113749    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c5d0357-a8f0-484c-9584-4b0184fd5284-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c5d0357-a8f0-484c-9584-4b0184fd5284-istio-data\") pod \"9c5d0357-a8f0-484c-9584-4b0184fd5284\" (UID: \"9c5d0357-a8f0-484c-9584-4b0184fd5284\") "
Oct 30 22:27:46.113963 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:46.113790    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c5d0357-a8f0-484c-9584-4b0184fd5284-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c5d0357-a8f0-484c-9584-4b0184fd5284-workload-socket\") pod \"9c5d0357-a8f0-484c-9584-4b0184fd5284\" (UID: \"9c5d0357-a8f0-484c-9584-4b0184fd5284\") "
Oct 30 22:27:46.113963 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:46.113850    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9c5d0357-a8f0-484c-9584-4b0184fd5284-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c5d0357-a8f0-484c-9584-4b0184fd5284-istio-podinfo\") pod \"9c5d0357-a8f0-484c-9584-4b0184fd5284\" (UID: \"9c5d0357-a8f0-484c-9584-4b0184fd5284\") "
Oct 30 22:27:46.113963 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:46.113910    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c5d0357-a8f0-484c-9584-4b0184fd5284-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c5d0357-a8f0-484c-9584-4b0184fd5284-istio-envoy\") pod \"9c5d0357-a8f0-484c-9584-4b0184fd5284\" (UID: \"9c5d0357-a8f0-484c-9584-4b0184fd5284\") "
Oct 30 22:27:46.113963 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:46.113940    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c5d0357-a8f0-484c-9584-4b0184fd5284-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c5d0357-a8f0-484c-9584-4b0184fd5284-credential-socket\") pod \"9c5d0357-a8f0-484c-9584-4b0184fd5284\" (UID: \"9c5d0357-a8f0-484c-9584-4b0184fd5284\") "
Oct 30 22:27:46.114172 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:46.114017    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c5d0357-a8f0-484c-9584-4b0184fd5284-workload-certs" pod "9c5d0357-a8f0-484c-9584-4b0184fd5284" (UID: "9c5d0357-a8f0-484c-9584-4b0184fd5284"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:27:46.114172 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:46.114044    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c5d0357-a8f0-484c-9584-4b0184fd5284-istiod-ca-cert" pod "9c5d0357-a8f0-484c-9584-4b0184fd5284" (UID: "9c5d0357-a8f0-484c-9584-4b0184fd5284"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:27:46.114172 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:46.114059    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c5d0357-a8f0-484c-9584-4b0184fd5284-istio-data" pod "9c5d0357-a8f0-484c-9584-4b0184fd5284" (UID: "9c5d0357-a8f0-484c-9584-4b0184fd5284"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:27:46.114328 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:46.114158    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c5d0357-a8f0-484c-9584-4b0184fd5284-workload-socket" pod "9c5d0357-a8f0-484c-9584-4b0184fd5284" (UID: "9c5d0357-a8f0-484c-9584-4b0184fd5284"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:27:46.114388 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:46.114346    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c5d0357-a8f0-484c-9584-4b0184fd5284-credential-socket" pod "9c5d0357-a8f0-484c-9584-4b0184fd5284" (UID: "9c5d0357-a8f0-484c-9584-4b0184fd5284"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:27:46.114691 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:46.114670    5678 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c5d0357-a8f0-484c-9584-4b0184fd5284-istiod-ca-cert\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:46.114809 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:46.114693    5678 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c5d0357-a8f0-484c-9584-4b0184fd5284-workload-certs\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:46.114809 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:46.114723    5678 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c5d0357-a8f0-484c-9584-4b0184fd5284-istio-data\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:46.114809 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:46.114737    5678 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c5d0357-a8f0-484c-9584-4b0184fd5284-workload-socket\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:46.114809 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:46.114750    5678 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c5d0357-a8f0-484c-9584-4b0184fd5284-credential-socket\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:46.116073 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:46.116045    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c5d0357-a8f0-484c-9584-4b0184fd5284-kube-api-access-s8kqp" pod "9c5d0357-a8f0-484c-9584-4b0184fd5284" (UID: "9c5d0357-a8f0-484c-9584-4b0184fd5284"). InnerVolumeSpecName "kube-api-access-s8kqp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:27:46.116183 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:46.116088    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c5d0357-a8f0-484c-9584-4b0184fd5284-istio-token" pod "9c5d0357-a8f0-484c-9584-4b0184fd5284" (UID: "9c5d0357-a8f0-484c-9584-4b0184fd5284"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:27:46.116394 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:46.116378    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c5d0357-a8f0-484c-9584-4b0184fd5284-istio-envoy" pod "9c5d0357-a8f0-484c-9584-4b0184fd5284" (UID: "9c5d0357-a8f0-484c-9584-4b0184fd5284"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:27:46.116471 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:46.116387    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9c5d0357-a8f0-484c-9584-4b0184fd5284-istio-podinfo" pod "9c5d0357-a8f0-484c-9584-4b0184fd5284" (UID: "9c5d0357-a8f0-484c-9584-4b0184fd5284"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:27:46.127435 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:46.127375    5678 generic.go:356] "Generic (PLEG): container finished" podID="9c5d0357-a8f0-484c-9584-4b0184fd5284" containerID="8a0e4e964afec05f0364932bdf0add52bb8c1557985001aa137064730590af54" exitCode=0
Oct 30 22:27:46.127529 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:46.127431    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5637a7-openshift-default-5b47575485-dxr9g" event={"ID":"9c5d0357-a8f0-484c-9584-4b0184fd5284","Type":"ContainerDied","Data":"8a0e4e964afec05f0364932bdf0add52bb8c1557985001aa137064730590af54"}
Oct 30 22:27:46.127529 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:46.127489    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5637a7-openshift-default-5b47575485-dxr9g" event={"ID":"9c5d0357-a8f0-484c-9584-4b0184fd5284","Type":"ContainerDied","Data":"83c1f197f9a6d0e20ae28b6809e9bdc55d8f9cdd09b6fc0f01cc33a288bce3df"}
Oct 30 22:27:46.127529 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:46.127495    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5637a7-openshift-default-5b47575485-dxr9g"
Oct 30 22:27:46.127529 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:46.127516    5678 scope.go:122] "RemoveContainer" containerID="8a0e4e964afec05f0364932bdf0add52bb8c1557985001aa137064730590af54"
Oct 30 22:27:46.146692 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:46.146671    5678 scope.go:122] "RemoveContainer" containerID="79480cc4e93133cbe7c6c8aa113e4499d86701e9e32b9d50abb75fa7606c27af"
Oct 30 22:27:46.158706 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:46.158678    5678 scope.go:122] "RemoveContainer" containerID="8a0e4e964afec05f0364932bdf0add52bb8c1557985001aa137064730590af54"
Oct 30 22:27:46.158978 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:27:46.158959    5678 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a0e4e964afec05f0364932bdf0add52bb8c1557985001aa137064730590af54\": container with ID starting with 8a0e4e964afec05f0364932bdf0add52bb8c1557985001aa137064730590af54 not found: ID does not exist" containerID="8a0e4e964afec05f0364932bdf0add52bb8c1557985001aa137064730590af54"
Oct 30 22:27:46.159042 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:46.158985    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a0e4e964afec05f0364932bdf0add52bb8c1557985001aa137064730590af54"} err="failed to get container status \"8a0e4e964afec05f0364932bdf0add52bb8c1557985001aa137064730590af54\": rpc error: code = NotFound desc = could not find container \"8a0e4e964afec05f0364932bdf0add52bb8c1557985001aa137064730590af54\": container with ID starting with 8a0e4e964afec05f0364932bdf0add52bb8c1557985001aa137064730590af54 not found: ID does not exist"
Oct 30 22:27:46.159042 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:46.159006    5678 scope.go:122] "RemoveContainer" containerID="79480cc4e93133cbe7c6c8aa113e4499d86701e9e32b9d50abb75fa7606c27af"
Oct 30 22:27:46.159264 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:27:46.159247    5678 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79480cc4e93133cbe7c6c8aa113e4499d86701e9e32b9d50abb75fa7606c27af\": container with ID starting with 79480cc4e93133cbe7c6c8aa113e4499d86701e9e32b9d50abb75fa7606c27af not found: ID does not exist" containerID="79480cc4e93133cbe7c6c8aa113e4499d86701e9e32b9d50abb75fa7606c27af"
Oct 30 22:27:46.159312 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:46.159269    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79480cc4e93133cbe7c6c8aa113e4499d86701e9e32b9d50abb75fa7606c27af"} err="failed to get container status \"79480cc4e93133cbe7c6c8aa113e4499d86701e9e32b9d50abb75fa7606c27af\": rpc error: code = NotFound desc = could not find container \"79480cc4e93133cbe7c6c8aa113e4499d86701e9e32b9d50abb75fa7606c27af\": container with ID starting with 79480cc4e93133cbe7c6c8aa113e4499d86701e9e32b9d50abb75fa7606c27af not found: ID does not exist"
Oct 30 22:27:46.179847 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:46.179795    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-595fd7d7-656b887dc9-5fgv6" podStartSLOduration=5.23764897 podStartE2EDuration="6.17977972s" totalImagesPullingTime="942.13075ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:27:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:27:44.230654792 +0000 UTC m=+2241.834962346" watchObservedRunningTime="2026-10-30 22:27:46.17977972 +0000 UTC m=+2243.784087273"
Oct 30 22:27:46.216417 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:46.216360    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8kqp\" (UniqueName: \"kubernetes.io/projected/9c5d0357-a8f0-484c-9584-4b0184fd5284-kube-api-access-s8kqp\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:46.216417 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:46.216395    5678 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c5d0357-a8f0-484c-9584-4b0184fd5284-istio-token\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:46.216417 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:46.216418    5678 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c5d0357-a8f0-484c-9584-4b0184fd5284-istio-podinfo\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:46.216724 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:46.216430    5678 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c5d0357-a8f0-484c-9584-4b0184fd5284-istio-envoy\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:46.286100 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:46.286066    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-5637a7-openshift-default-5b47575485-dxr9g"]
Oct 30 22:27:46.292991 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:46.292918    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-5637a7-openshift-default-5b47575485-dxr9g"]
Oct 30 22:27:46.529178 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:46.529146    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c5d0357-a8f0-484c-9584-4b0184fd5284" path="/var/lib/kubelet/pods/9c5d0357-a8f0-484c-9584-4b0184fd5284/volumes"
Oct 30 22:27:47.702339 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:47.702303    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-7bb1c2/distinct-w3-7bb1c2-kserve-8d695596c-8zgtw"]
Oct 30 22:27:47.749015 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:47.748986    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-7bb1c2/premium-w3-7bb1c2-kserve-7dbf875d88-dqxn5"]
Oct 30 22:27:47.752622 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:47.752595    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-7bb1c2/simulator-w3-7bb1c2-kserve-675645869c-vcwqv"]
Oct 30 22:27:50.122453 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:27:50.122386    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-595fd7d7-656b887dc9-5fgv6"
Oct 30 22:28:02.724214 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:02.724132    5678 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-7bb1c2/distinct-w3-7bb1c2-kserve-8d695596c-8zgtw" podUID="36f0a6a2-54be-49bf-9202-502960cbb134" containerName="main" containerID="cri-o://bd70d8c9ea9e84fb68044057299853590dd9d43ce9575a0bab5a3ba895b8f503" gracePeriod=45
Oct 30 22:28:02.774222 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:02.773381    5678 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-7bb1c2/premium-w3-7bb1c2-kserve-7dbf875d88-dqxn5" podUID="a58c911f-e2eb-4820-a6b6-24afe78b5fd4" containerName="main" containerID="cri-o://847e616c8cb13638a02400802170fbea1247c1ac01ba3d3d06107f923c6060fd" gracePeriod=45
Oct 30 22:28:02.776818 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:02.776769    5678 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-7bb1c2/simulator-w3-7bb1c2-kserve-675645869c-vcwqv" podUID="1fce9daf-68c9-41d0-825b-2012bb93c8ba" containerName="main" containerID="cri-o://864e0d7ba3a5a06cc6ef94c2d50658438a65d46b3fb3de3d21181d619a78c121" gracePeriod=45
Oct 30 22:28:02.911504 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:02.911137    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7bb1c2/simulator-w3-7bb1c2-kserve-675645869c-vcwqv" podUID="1fce9daf-68c9-41d0-825b-2012bb93c8ba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": dial tcp 10.134.0.54:8000: connect: connection refused"
Oct 30 22:28:03.096849 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:03.096824    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-7bb1c2/distinct-w3-7bb1c2-kserve-8d695596c-8zgtw"
Oct 30 22:28:03.098812 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:03.098792    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-7bb1c2/simulator-w3-7bb1c2-kserve-675645869c-vcwqv"
Oct 30 22:28:03.103920 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:03.103897    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-7bb1c2/premium-w3-7bb1c2-kserve-7dbf875d88-dqxn5"
Oct 30 22:28:03.194241 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:03.194208    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36f0a6a2-54be-49bf-9202-502960cbb134-dshm\" (UniqueName: \"kubernetes.io/empty-dir/36f0a6a2-54be-49bf-9202-502960cbb134-dshm\") pod \"36f0a6a2-54be-49bf-9202-502960cbb134\" (UID: \"36f0a6a2-54be-49bf-9202-502960cbb134\") "
Oct 30 22:28:03.194448 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:03.194251    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a58c911f-e2eb-4820-a6b6-24afe78b5fd4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a58c911f-e2eb-4820-a6b6-24afe78b5fd4-tmp-dir\") pod \"a58c911f-e2eb-4820-a6b6-24afe78b5fd4\" (UID: \"a58c911f-e2eb-4820-a6b6-24afe78b5fd4\") "
Oct 30 22:28:03.194448 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:03.194291    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a58c911f-e2eb-4820-a6b6-24afe78b5fd4-tls-certs\" (UniqueName: \"kubernetes.io/secret/a58c911f-e2eb-4820-a6b6-24afe78b5fd4-tls-certs\") pod \"a58c911f-e2eb-4820-a6b6-24afe78b5fd4\" (UID: \"a58c911f-e2eb-4820-a6b6-24afe78b5fd4\") "
Oct 30 22:28:03.194448 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:03.194350    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a58c911f-e2eb-4820-a6b6-24afe78b5fd4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a58c911f-e2eb-4820-a6b6-24afe78b5fd4-dshm\") pod \"a58c911f-e2eb-4820-a6b6-24afe78b5fd4\" (UID: \"a58c911f-e2eb-4820-a6b6-24afe78b5fd4\") "
Oct 30 22:28:03.194448 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:03.194381    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36f0a6a2-54be-49bf-9202-502960cbb134-tls-certs\" (UniqueName: \"kubernetes.io/secret/36f0a6a2-54be-49bf-9202-502960cbb134-tls-certs\") pod \"36f0a6a2-54be-49bf-9202-502960cbb134\" (UID: \"36f0a6a2-54be-49bf-9202-502960cbb134\") "
Oct 30 22:28:03.194705 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:03.194478    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a58c911f-e2eb-4820-a6b6-24afe78b5fd4-home\" (UniqueName: \"kubernetes.io/empty-dir/a58c911f-e2eb-4820-a6b6-24afe78b5fd4-home\") pod \"a58c911f-e2eb-4820-a6b6-24afe78b5fd4\" (UID: \"a58c911f-e2eb-4820-a6b6-24afe78b5fd4\") "
Oct 30 22:28:03.194705 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:03.194542    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a58c911f-e2eb-4820-a6b6-24afe78b5fd4-kube-api-access-xvft7\" (UniqueName: \"kubernetes.io/projected/a58c911f-e2eb-4820-a6b6-24afe78b5fd4-kube-api-access-xvft7\") pod \"a58c911f-e2eb-4820-a6b6-24afe78b5fd4\" (UID: \"a58c911f-e2eb-4820-a6b6-24afe78b5fd4\") "
Oct 30 22:28:03.194705 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:03.194570    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36f0a6a2-54be-49bf-9202-502960cbb134-kube-api-access-wcfq5\" (UniqueName: \"kubernetes.io/projected/36f0a6a2-54be-49bf-9202-502960cbb134-kube-api-access-wcfq5\") pod \"36f0a6a2-54be-49bf-9202-502960cbb134\" (UID: \"36f0a6a2-54be-49bf-9202-502960cbb134\") "
Oct 30 22:28:03.194705 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:03.194578    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a58c911f-e2eb-4820-a6b6-24afe78b5fd4-tmp-dir" pod "a58c911f-e2eb-4820-a6b6-24afe78b5fd4" (UID: "a58c911f-e2eb-4820-a6b6-24afe78b5fd4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:28:03.194705 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:03.194607    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fce9daf-68c9-41d0-825b-2012bb93c8ba-home\" (UniqueName: \"kubernetes.io/empty-dir/1fce9daf-68c9-41d0-825b-2012bb93c8ba-home\") pod \"1fce9daf-68c9-41d0-825b-2012bb93c8ba\" (UID: \"1fce9daf-68c9-41d0-825b-2012bb93c8ba\") "
Oct 30 22:28:03.194705 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:03.194655    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1fce9daf-68c9-41d0-825b-2012bb93c8ba-tls-certs\" (UniqueName: \"kubernetes.io/secret/1fce9daf-68c9-41d0-825b-2012bb93c8ba-tls-certs\") pod \"1fce9daf-68c9-41d0-825b-2012bb93c8ba\" (UID: \"1fce9daf-68c9-41d0-825b-2012bb93c8ba\") "
Oct 30 22:28:03.195040 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:03.194704    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fce9daf-68c9-41d0-825b-2012bb93c8ba-kube-api-access-8tf48\" (UniqueName: \"kubernetes.io/projected/1fce9daf-68c9-41d0-825b-2012bb93c8ba-kube-api-access-8tf48\") pod \"1fce9daf-68c9-41d0-825b-2012bb93c8ba\" (UID: \"1fce9daf-68c9-41d0-825b-2012bb93c8ba\") "
Oct 30 22:28:03.195040 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:03.194757    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a58c911f-e2eb-4820-a6b6-24afe78b5fd4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a58c911f-e2eb-4820-a6b6-24afe78b5fd4-model-cache\") pod \"a58c911f-e2eb-4820-a6b6-24afe78b5fd4\" (UID: \"a58c911f-e2eb-4820-a6b6-24afe78b5fd4\") "
Oct 30 22:28:03.195040 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:03.194821    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36f0a6a2-54be-49bf-9202-502960cbb134-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36f0a6a2-54be-49bf-9202-502960cbb134-model-cache\") pod \"36f0a6a2-54be-49bf-9202-502960cbb134\" (UID: \"36f0a6a2-54be-49bf-9202-502960cbb134\") "
Oct 30 22:28:03.195040 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:03.194848    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fce9daf-68c9-41d0-825b-2012bb93c8ba-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fce9daf-68c9-41d0-825b-2012bb93c8ba-dshm\") pod \"1fce9daf-68c9-41d0-825b-2012bb93c8ba\" (UID: \"1fce9daf-68c9-41d0-825b-2012bb93c8ba\") "
Oct 30 22:28:03.195040 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:03.194868    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a58c911f-e2eb-4820-a6b6-24afe78b5fd4-home" pod "a58c911f-e2eb-4820-a6b6-24afe78b5fd4" (UID: "a58c911f-e2eb-4820-a6b6-24afe78b5fd4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:28:03.195040 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:03.194894    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fce9daf-68c9-41d0-825b-2012bb93c8ba-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fce9daf-68c9-41d0-825b-2012bb93c8ba-model-cache\") pod \"1fce9daf-68c9-41d0-825b-2012bb93c8ba\" (UID: \"1fce9daf-68c9-41d0-825b-2012bb93c8ba\") "
Oct 30 22:28:03.195040 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:03.195011    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36f0a6a2-54be-49bf-9202-502960cbb134-home\" (UniqueName: \"kubernetes.io/empty-dir/36f0a6a2-54be-49bf-9202-502960cbb134-home\") pod \"36f0a6a2-54be-49bf-9202-502960cbb134\" (UID: \"36f0a6a2-54be-49bf-9202-502960cbb134\") "
Oct 30 22:28:03.195432 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:03.195053    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36f0a6a2-54be-49bf-9202-502960cbb134-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36f0a6a2-54be-49bf-9202-502960cbb134-tmp-dir\") pod \"36f0a6a2-54be-49bf-9202-502960cbb134\" (UID: \"36f0a6a2-54be-49bf-9202-502960cbb134\") "
Oct 30 22:28:03.195432 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:03.195129    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fce9daf-68c9-41d0-825b-2012bb93c8ba-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fce9daf-68c9-41d0-825b-2012bb93c8ba-tmp-dir\") pod \"1fce9daf-68c9-41d0-825b-2012bb93c8ba\" (UID: \"1fce9daf-68c9-41d0-825b-2012bb93c8ba\") "
Oct 30 22:28:03.195557 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:03.195473    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fce9daf-68c9-41d0-825b-2012bb93c8ba-home" pod "1fce9daf-68c9-41d0-825b-2012bb93c8ba" (UID: "1fce9daf-68c9-41d0-825b-2012bb93c8ba"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:28:03.195708 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:03.195683    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fce9daf-68c9-41d0-825b-2012bb93c8ba-tmp-dir" pod "1fce9daf-68c9-41d0-825b-2012bb93c8ba" (UID: "1fce9daf-68c9-41d0-825b-2012bb93c8ba"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:28:03.195972 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:03.195951    5678 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1fce9daf-68c9-41d0-825b-2012bb93c8ba-home\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:03.196052 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:03.195976    5678 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fce9daf-68c9-41d0-825b-2012bb93c8ba-tmp-dir\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:03.196052 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:03.195990    5678 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a58c911f-e2eb-4820-a6b6-24afe78b5fd4-tmp-dir\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:03.196052 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:03.196009    5678 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a58c911f-e2eb-4820-a6b6-24afe78b5fd4-home\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:03.196225 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:03.196196    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36f0a6a2-54be-49bf-9202-502960cbb134-home" pod "36f0a6a2-54be-49bf-9202-502960cbb134" (UID: "36f0a6a2-54be-49bf-9202-502960cbb134"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:28:03.196288 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:03.196216    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a58c911f-e2eb-4820-a6b6-24afe78b5fd4-model-cache" pod "a58c911f-e2eb-4820-a6b6-24afe78b5fd4" (UID: "a58c911f-e2eb-4820-a6b6-24afe78b5fd4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:28:03.196288 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:03.196223    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36f0a6a2-54be-49bf-9202-502960cbb134-tmp-dir" pod "36f0a6a2-54be-49bf-9202-502960cbb134" (UID: "36f0a6a2-54be-49bf-9202-502960cbb134"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:28:03.196389 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:03.196288    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36f0a6a2-54be-49bf-9202-502960cbb134-model-cache" pod "36f0a6a2-54be-49bf-9202-502960cbb134" (UID: "36f0a6a2-54be-49bf-9202-502960cbb134"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:28:03.196472 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:03.196383    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fce9daf-68c9-41d0-825b-2012bb93c8ba-model-cache" pod "1fce9daf-68c9-41d0-825b-2012bb93c8ba" (UID: "1fce9daf-68c9-41d0-825b-2012bb93c8ba"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:28:03.198380 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:03.198358    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fce9daf-68c9-41d0-825b-2012bb93c8ba-dshm" pod "1fce9daf-68c9-41d0-825b-2012bb93c8ba" (UID: "1fce9daf-68c9-41d0-825b-2012bb93c8ba"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:28:03.198380 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:03.198370    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36f0a6a2-54be-49bf-9202-502960cbb134-tls-certs" pod "36f0a6a2-54be-49bf-9202-502960cbb134" (UID: "36f0a6a2-54be-49bf-9202-502960cbb134"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:28:03.198549 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:03.198379    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a58c911f-e2eb-4820-a6b6-24afe78b5fd4-dshm" pod "a58c911f-e2eb-4820-a6b6-24afe78b5fd4" (UID: "a58c911f-e2eb-4820-a6b6-24afe78b5fd4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:28:03.198549 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:03.198429    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36f0a6a2-54be-49bf-9202-502960cbb134-kube-api-access-wcfq5" pod "36f0a6a2-54be-49bf-9202-502960cbb134" (UID: "36f0a6a2-54be-49bf-9202-502960cbb134"). InnerVolumeSpecName "kube-api-access-wcfq5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:28:03.198549 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:03.198434    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a58c911f-e2eb-4820-a6b6-24afe78b5fd4-tls-certs" pod "a58c911f-e2eb-4820-a6b6-24afe78b5fd4" (UID: "a58c911f-e2eb-4820-a6b6-24afe78b5fd4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:28:03.198549 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:03.198458    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36f0a6a2-54be-49bf-9202-502960cbb134-dshm" pod "36f0a6a2-54be-49bf-9202-502960cbb134" (UID: "36f0a6a2-54be-49bf-9202-502960cbb134"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:28:03.198549 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:03.198474    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a58c911f-e2eb-4820-a6b6-24afe78b5fd4-kube-api-access-xvft7" pod "a58c911f-e2eb-4820-a6b6-24afe78b5fd4" (UID: "a58c911f-e2eb-4820-a6b6-24afe78b5fd4"). InnerVolumeSpecName "kube-api-access-xvft7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:28:03.198708 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:03.198666    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fce9daf-68c9-41d0-825b-2012bb93c8ba-kube-api-access-8tf48" pod "1fce9daf-68c9-41d0-825b-2012bb93c8ba" (UID: "1fce9daf-68c9-41d0-825b-2012bb93c8ba"). InnerVolumeSpecName "kube-api-access-8tf48". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:28:03.199202 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:03.199176    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fce9daf-68c9-41d0-825b-2012bb93c8ba-tls-certs" pod "1fce9daf-68c9-41d0-825b-2012bb93c8ba" (UID: "1fce9daf-68c9-41d0-825b-2012bb93c8ba"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:28:03.263138 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:03.263052    5678 generic.go:356] "Generic (PLEG): container finished" podID="a58c911f-e2eb-4820-a6b6-24afe78b5fd4" containerID="847e616c8cb13638a02400802170fbea1247c1ac01ba3d3d06107f923c6060fd" exitCode=0
Oct 30 22:28:03.263138 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:03.263103    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-7bb1c2/premium-w3-7bb1c2-kserve-7dbf875d88-dqxn5"
Oct 30 22:28:03.263378 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:03.263215    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-7bb1c2/premium-w3-7bb1c2-kserve-7dbf875d88-dqxn5" event={"ID":"a58c911f-e2eb-4820-a6b6-24afe78b5fd4","Type":"ContainerDied","Data":"847e616c8cb13638a02400802170fbea1247c1ac01ba3d3d06107f923c6060fd"}
Oct 30 22:28:03.263378 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:03.263245    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-7bb1c2/premium-w3-7bb1c2-kserve-7dbf875d88-dqxn5" event={"ID":"a58c911f-e2eb-4820-a6b6-24afe78b5fd4","Type":"ContainerDied","Data":"7db2ee57a6bc5b2ab0c3b1f82c03272077341767e8dd93b3a1eb8fdac07a848f"}
Oct 30 22:28:03.263378 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:03.263261    5678 scope.go:122] "RemoveContainer" containerID="847e616c8cb13638a02400802170fbea1247c1ac01ba3d3d06107f923c6060fd"
Oct 30 22:28:03.265530 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:03.265142    5678 generic.go:356] "Generic (PLEG): container finished" podID="1fce9daf-68c9-41d0-825b-2012bb93c8ba" containerID="864e0d7ba3a5a06cc6ef94c2d50658438a65d46b3fb3de3d21181d619a78c121" exitCode=0
Oct 30 22:28:03.265530 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:03.265182    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-7bb1c2/simulator-w3-7bb1c2-kserve-675645869c-vcwqv"
Oct 30 22:28:03.265530 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:03.265364    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-7bb1c2/simulator-w3-7bb1c2-kserve-675645869c-vcwqv" event={"ID":"1fce9daf-68c9-41d0-825b-2012bb93c8ba","Type":"ContainerDied","Data":"864e0d7ba3a5a06cc6ef94c2d50658438a65d46b3fb3de3d21181d619a78c121"}
Oct 30 22:28:03.265530 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:03.265494    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-7bb1c2/simulator-w3-7bb1c2-kserve-675645869c-vcwqv" event={"ID":"1fce9daf-68c9-41d0-825b-2012bb93c8ba","Type":"ContainerDied","Data":"9c659b5bb1063703e4eb6bb683994042dc86e9890c2e004c5dbc84e0e50313da"}
Oct 30 22:28:03.277447 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:03.277429    5678 scope.go:122] "RemoveContainer" containerID="847e616c8cb13638a02400802170fbea1247c1ac01ba3d3d06107f923c6060fd"
Oct 30 22:28:03.277766 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:28:03.277740    5678 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"847e616c8cb13638a02400802170fbea1247c1ac01ba3d3d06107f923c6060fd\": container with ID starting with 847e616c8cb13638a02400802170fbea1247c1ac01ba3d3d06107f923c6060fd not found: ID does not exist" containerID="847e616c8cb13638a02400802170fbea1247c1ac01ba3d3d06107f923c6060fd"
Oct 30 22:28:03.277889 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:03.277776    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"847e616c8cb13638a02400802170fbea1247c1ac01ba3d3d06107f923c6060fd"} err="failed to get container status \"847e616c8cb13638a02400802170fbea1247c1ac01ba3d3d06107f923c6060fd\": rpc error: code = NotFound desc = could not find container \"847e616c8cb13638a02400802170fbea1247c1ac01ba3d3d06107f923c6060fd\": container with ID starting with 847e616c8cb13638a02400802170fbea1247c1ac01ba3d3d06107f923c6060fd not found: ID does not exist"
Oct 30 22:28:03.277889 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:03.277802    5678 scope.go:122] "RemoveContainer" containerID="864e0d7ba3a5a06cc6ef94c2d50658438a65d46b3fb3de3d21181d619a78c121"
Oct 30 22:28:03.278103 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:03.278083    5678 generic.go:356] "Generic (PLEG): container finished" podID="36f0a6a2-54be-49bf-9202-502960cbb134" containerID="bd70d8c9ea9e84fb68044057299853590dd9d43ce9575a0bab5a3ba895b8f503" exitCode=0
Oct 30 22:28:03.278181 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:03.278124    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-7bb1c2/distinct-w3-7bb1c2-kserve-8d695596c-8zgtw" event={"ID":"36f0a6a2-54be-49bf-9202-502960cbb134","Type":"ContainerDied","Data":"bd70d8c9ea9e84fb68044057299853590dd9d43ce9575a0bab5a3ba895b8f503"}
Oct 30 22:28:03.278181 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:03.278150    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-7bb1c2/distinct-w3-7bb1c2-kserve-8d695596c-8zgtw" event={"ID":"36f0a6a2-54be-49bf-9202-502960cbb134","Type":"ContainerDied","Data":"9fbfaf334ea0832a94b4c4d520b277f8eb1d1487462328dbfadcab2747fe9454"}
Oct 30 22:28:03.278253 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:03.278192    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-7bb1c2/distinct-w3-7bb1c2-kserve-8d695596c-8zgtw"
Oct 30 22:28:03.287867 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:03.287849    5678 scope.go:122] "RemoveContainer" containerID="864e0d7ba3a5a06cc6ef94c2d50658438a65d46b3fb3de3d21181d619a78c121"
Oct 30 22:28:03.288126 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:28:03.288104    5678 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"864e0d7ba3a5a06cc6ef94c2d50658438a65d46b3fb3de3d21181d619a78c121\": container with ID starting with 864e0d7ba3a5a06cc6ef94c2d50658438a65d46b3fb3de3d21181d619a78c121 not found: ID does not exist" containerID="864e0d7ba3a5a06cc6ef94c2d50658438a65d46b3fb3de3d21181d619a78c121"
Oct 30 22:28:03.288178 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:03.288135    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"864e0d7ba3a5a06cc6ef94c2d50658438a65d46b3fb3de3d21181d619a78c121"} err="failed to get container status \"864e0d7ba3a5a06cc6ef94c2d50658438a65d46b3fb3de3d21181d619a78c121\": rpc error: code = NotFound desc = could not find container \"864e0d7ba3a5a06cc6ef94c2d50658438a65d46b3fb3de3d21181d619a78c121\": container with ID starting with 864e0d7ba3a5a06cc6ef94c2d50658438a65d46b3fb3de3d21181d619a78c121 not found: ID does not exist"
Oct 30 22:28:03.288178 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:03.288155    5678 scope.go:122] "RemoveContainer" containerID="bd70d8c9ea9e84fb68044057299853590dd9d43ce9575a0bab5a3ba895b8f503"
Oct 30 22:28:03.297050 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:03.297033    5678 scope.go:122] "RemoveContainer" containerID="bd70d8c9ea9e84fb68044057299853590dd9d43ce9575a0bab5a3ba895b8f503"
Oct 30 22:28:03.297152 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:03.297133    5678 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a58c911f-e2eb-4820-a6b6-24afe78b5fd4-tls-certs\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:03.297221 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:03.297161    5678 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a58c911f-e2eb-4820-a6b6-24afe78b5fd4-dshm\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:03.297221 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:03.297177    5678 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36f0a6a2-54be-49bf-9202-502960cbb134-tls-certs\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:03.297221 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:03.297191    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvft7\" (UniqueName: \"kubernetes.io/projected/a58c911f-e2eb-4820-a6b6-24afe78b5fd4-kube-api-access-xvft7\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:03.297221 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:03.297205    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcfq5\" (UniqueName: \"kubernetes.io/projected/36f0a6a2-54be-49bf-9202-502960cbb134-kube-api-access-wcfq5\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:03.297393 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:03.297220    5678 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fce9daf-68c9-41d0-825b-2012bb93c8ba-tls-certs\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:03.297393 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:03.297234    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tf48\" (UniqueName: \"kubernetes.io/projected/1fce9daf-68c9-41d0-825b-2012bb93c8ba-kube-api-access-8tf48\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:03.297393 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:03.297248    5678 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a58c911f-e2eb-4820-a6b6-24afe78b5fd4-model-cache\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:03.297393 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:03.297262    5678 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36f0a6a2-54be-49bf-9202-502960cbb134-model-cache\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:03.297393 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:03.297275    5678 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fce9daf-68c9-41d0-825b-2012bb93c8ba-dshm\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:03.297393 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:03.297288    5678 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fce9daf-68c9-41d0-825b-2012bb93c8ba-model-cache\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:03.297393 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:28:03.297294    5678 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd70d8c9ea9e84fb68044057299853590dd9d43ce9575a0bab5a3ba895b8f503\": container with ID starting with bd70d8c9ea9e84fb68044057299853590dd9d43ce9575a0bab5a3ba895b8f503 not found: ID does not exist" containerID="bd70d8c9ea9e84fb68044057299853590dd9d43ce9575a0bab5a3ba895b8f503"
Oct 30 22:28:03.297393 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:03.297317    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd70d8c9ea9e84fb68044057299853590dd9d43ce9575a0bab5a3ba895b8f503"} err="failed to get container status \"bd70d8c9ea9e84fb68044057299853590dd9d43ce9575a0bab5a3ba895b8f503\": rpc error: code = NotFound desc = could not find container \"bd70d8c9ea9e84fb68044057299853590dd9d43ce9575a0bab5a3ba895b8f503\": container with ID starting with bd70d8c9ea9e84fb68044057299853590dd9d43ce9575a0bab5a3ba895b8f503 not found: ID does not exist"
Oct 30 22:28:03.297393 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:03.297302    5678 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36f0a6a2-54be-49bf-9202-502960cbb134-home\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:03.297393 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:03.297342    5678 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36f0a6a2-54be-49bf-9202-502960cbb134-tmp-dir\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:03.297393 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:03.297351    5678 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36f0a6a2-54be-49bf-9202-502960cbb134-dshm\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:03.546570 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:03.546488    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-7bb1c2/distinct-w3-7bb1c2-kserve-8d695596c-8zgtw"]
Oct 30 22:28:03.557421 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:03.557365    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-7bb1c2/distinct-w3-7bb1c2-kserve-8d695596c-8zgtw"]
Oct 30 22:28:03.658994 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:03.658959    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-7bb1c2/simulator-w3-7bb1c2-kserve-675645869c-vcwqv"]
Oct 30 22:28:03.663204 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:03.663179    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-7bb1c2/simulator-w3-7bb1c2-kserve-675645869c-vcwqv"]
Oct 30 22:28:03.767983 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:03.767957    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-7bb1c2/premium-w3-7bb1c2-kserve-7dbf875d88-dqxn5"]
Oct 30 22:28:03.772556 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:03.772534    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-7bb1c2/premium-w3-7bb1c2-kserve-7dbf875d88-dqxn5"]
Oct 30 22:28:04.043020 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:04.042981    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7bb1c2/premium-w3-7bb1c2-kserve-7dbf875d88-dqxn5" podUID="a58c911f-e2eb-4820-a6b6-24afe78b5fd4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:28:04.524126 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:04.524094    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1fce9daf-68c9-41d0-825b-2012bb93c8ba" path="/var/lib/kubelet/pods/1fce9daf-68c9-41d0-825b-2012bb93c8ba/volumes"
Oct 30 22:28:04.524653 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:04.524637    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36f0a6a2-54be-49bf-9202-502960cbb134" path="/var/lib/kubelet/pods/36f0a6a2-54be-49bf-9202-502960cbb134/volumes"
Oct 30 22:28:04.525053 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:04.525040    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a58c911f-e2eb-4820-a6b6-24afe78b5fd4" path="/var/lib/kubelet/pods/a58c911f-e2eb-4820-a6b6-24afe78b5fd4/volumes"
Oct 30 22:28:10.578592 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:28:10.578550    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 22:28:19.448635 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:19.448603    5678 generic.go:356] "Generic (PLEG): container finished" podID="707b0553-ce35-4de0-ba4a-fffa4adfc6ee" containerID="1dc97660aff8ac9223122df34c70ae1232b4e4ba5e5378e1b72098e2e8e770ec" exitCode=0
Oct 30 22:28:19.449190 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:19.448676    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-5686cb-openshift-default-5f6d8dbdbc-s8jnc" event={"ID":"707b0553-ce35-4de0-ba4a-fffa4adfc6ee","Type":"ContainerDied","Data":"1dc97660aff8ac9223122df34c70ae1232b4e4ba5e5378e1b72098e2e8e770ec"}
Oct 30 22:28:19.449280 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:19.449214    5678 scope.go:122] "RemoveContainer" containerID="1dc97660aff8ac9223122df34c70ae1232b4e4ba5e5378e1b72098e2e8e770ec"
Oct 30 22:28:19.451178 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:19.451152    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:28:19.451255 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:19.451232    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:28:19.451298 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:19.451265    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:28:20.462121 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:20.462082    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-5686cb-openshift-default-5f6d8dbdbc-s8jnc" event={"ID":"707b0553-ce35-4de0-ba4a-fffa4adfc6ee","Type":"ContainerStarted","Data":"84dba6a3fd54fd611ac8afeb27d450cd35128264db4e2356b46dee14fec5a6d0"}
Oct 30 22:28:20.646426 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:20.645844    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-5686cb-openshift-default-5f6d8dbdbc-s8jnc"
Oct 30 22:28:20.646426 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:20.645956    5678 patch_prober.go:29] interesting pod/e2e-api-b-5686cb-openshift-default-5f6d8dbdbc-s8jnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body=
Oct 30 22:28:20.646426 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:20.646005    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-5686cb-openshift-default-5f6d8dbdbc-s8jnc" podUID="707b0553-ce35-4de0-ba4a-fffa4adfc6ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused"
Oct 30 22:28:21.631424 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:21.628765    5678 patch_prober.go:29] interesting pod/e2e-api-b-5686cb-openshift-default-5f6d8dbdbc-s8jnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body=
Oct 30 22:28:21.631424 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:21.628826    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-5686cb-openshift-default-5f6d8dbdbc-s8jnc" podUID="707b0553-ce35-4de0-ba4a-fffa4adfc6ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused"
Oct 30 22:28:22.642786 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:22.642751    5678 patch_prober.go:29] interesting pod/e2e-api-b-5686cb-openshift-default-5f6d8dbdbc-s8jnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body=
Oct 30 22:28:22.643891 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:22.643858    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-5686cb-openshift-default-5f6d8dbdbc-s8jnc" podUID="707b0553-ce35-4de0-ba4a-fffa4adfc6ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused"
Oct 30 22:28:23.646925 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:23.646882    5678 patch_prober.go:29] interesting pod/e2e-api-b-5686cb-openshift-default-5f6d8dbdbc-s8jnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body=
Oct 30 22:28:23.647351 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:23.646962    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-5686cb-openshift-default-5f6d8dbdbc-s8jnc" podUID="707b0553-ce35-4de0-ba4a-fffa4adfc6ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused"
Oct 30 22:28:23.750570 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:23.750539    5678 scope.go:122] "RemoveContainer" containerID="a13691238de04f3734df5801600f4c12527d5438c2227cab25e360d5c3adde48"
Oct 30 22:28:23.760666 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:23.760644    5678 scope.go:122] "RemoveContainer" containerID="16f17f2328b48909248b4edd4484d4ad73959ad20cbb3a8a73f946b4e86ce39d"
Oct 30 22:28:24.634426 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:24.627786    5678 patch_prober.go:29] interesting pod/e2e-api-b-5686cb-openshift-default-5f6d8dbdbc-s8jnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body=
Oct 30 22:28:24.634426 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:24.627842    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-5686cb-openshift-default-5f6d8dbdbc-s8jnc" podUID="707b0553-ce35-4de0-ba4a-fffa4adfc6ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused"
Oct 30 22:28:25.627639 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:25.627606    5678 patch_prober.go:29] interesting pod/e2e-api-b-5686cb-openshift-default-5f6d8dbdbc-s8jnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body=
Oct 30 22:28:25.628132 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:25.627659    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-5686cb-openshift-default-5f6d8dbdbc-s8jnc" podUID="707b0553-ce35-4de0-ba4a-fffa4adfc6ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused"
Oct 30 22:28:26.631747 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:26.631710    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-5686cb-openshift-default-5f6d8dbdbc-s8jnc"
Oct 30 22:28:26.632238 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:26.632079    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-5686cb-openshift-default-5f6d8dbdbc-s8jnc"
Oct 30 22:28:26.632918 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:26.632899    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-5686cb-openshift-default-5f6d8dbdbc-s8jnc"
Oct 30 22:28:29.449222 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:29.449180    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-c24d61-openshift-default-7c56bd489-c85l8"]
Oct 30 22:28:29.449761 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:29.449504    5678 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-c24d61-openshift-default-7c56bd489-c85l8" podUID="bd8986aa-b616-4a9e-9dd9-ae7eefd3e847" containerName="istio-proxy" containerID="cri-o://69ca4fb4a4f0d20bb76fac59352f175389dd3e67b6534374f201b30d9c3999ba" gracePeriod=30
Oct 30 22:28:34.570094 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:34.570061    5678 generic.go:356] "Generic (PLEG): container finished" podID="bd8986aa-b616-4a9e-9dd9-ae7eefd3e847" containerID="69ca4fb4a4f0d20bb76fac59352f175389dd3e67b6534374f201b30d9c3999ba" exitCode=0
Oct 30 22:28:34.570556 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:34.570130    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c24d61-openshift-default-7c56bd489-c85l8" event={"ID":"bd8986aa-b616-4a9e-9dd9-ae7eefd3e847","Type":"ContainerDied","Data":"69ca4fb4a4f0d20bb76fac59352f175389dd3e67b6534374f201b30d9c3999ba"}
Oct 30 22:28:34.570556 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:34.570187    5678 scope.go:122] "RemoveContainer" containerID="8bed24e326a634e2ce195cb92ff0d418033527731d0cb1f61ca976b2d01ce2af"
Oct 30 22:28:34.856861 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:34.856829    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c24d61-openshift-default-7c56bd489-c85l8"
Oct 30 22:28:34.943532 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:34.943483    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd8986aa-b616-4a9e-9dd9-ae7eefd3e847-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd8986aa-b616-4a9e-9dd9-ae7eefd3e847-workload-socket\") pod \"bd8986aa-b616-4a9e-9dd9-ae7eefd3e847\" (UID: \"bd8986aa-b616-4a9e-9dd9-ae7eefd3e847\") "
Oct 30 22:28:34.944196 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:34.944056    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd8986aa-b616-4a9e-9dd9-ae7eefd3e847-workload-socket" pod "bd8986aa-b616-4a9e-9dd9-ae7eefd3e847" (UID: "bd8986aa-b616-4a9e-9dd9-ae7eefd3e847"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:28:34.944196 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:34.944185    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bd8986aa-b616-4a9e-9dd9-ae7eefd3e847-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd8986aa-b616-4a9e-9dd9-ae7eefd3e847-istio-podinfo\") pod \"bd8986aa-b616-4a9e-9dd9-ae7eefd3e847\" (UID: \"bd8986aa-b616-4a9e-9dd9-ae7eefd3e847\") "
Oct 30 22:28:34.944439 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:34.944346    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd8986aa-b616-4a9e-9dd9-ae7eefd3e847-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd8986aa-b616-4a9e-9dd9-ae7eefd3e847-istio-data\") pod \"bd8986aa-b616-4a9e-9dd9-ae7eefd3e847\" (UID: \"bd8986aa-b616-4a9e-9dd9-ae7eefd3e847\") "
Oct 30 22:28:34.944521 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:34.944462    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd8986aa-b616-4a9e-9dd9-ae7eefd3e847-kube-api-access-cp5tl\" (UniqueName: \"kubernetes.io/projected/bd8986aa-b616-4a9e-9dd9-ae7eefd3e847-kube-api-access-cp5tl\") pod \"bd8986aa-b616-4a9e-9dd9-ae7eefd3e847\" (UID: \"bd8986aa-b616-4a9e-9dd9-ae7eefd3e847\") "
Oct 30 22:28:34.944615 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:34.944583    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd8986aa-b616-4a9e-9dd9-ae7eefd3e847-istio-token\" (UniqueName: \"kubernetes.io/projected/bd8986aa-b616-4a9e-9dd9-ae7eefd3e847-istio-token\") pod \"bd8986aa-b616-4a9e-9dd9-ae7eefd3e847\" (UID: \"bd8986aa-b616-4a9e-9dd9-ae7eefd3e847\") "
Oct 30 22:28:34.944798 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:34.944781    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd8986aa-b616-4a9e-9dd9-ae7eefd3e847-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd8986aa-b616-4a9e-9dd9-ae7eefd3e847-istiod-ca-cert\") pod \"bd8986aa-b616-4a9e-9dd9-ae7eefd3e847\" (UID: \"bd8986aa-b616-4a9e-9dd9-ae7eefd3e847\") "
Oct 30 22:28:34.944982 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:34.944967    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd8986aa-b616-4a9e-9dd9-ae7eefd3e847-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd8986aa-b616-4a9e-9dd9-ae7eefd3e847-credential-socket\") pod \"bd8986aa-b616-4a9e-9dd9-ae7eefd3e847\" (UID: \"bd8986aa-b616-4a9e-9dd9-ae7eefd3e847\") "
Oct 30 22:28:34.945156 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:34.945099    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd8986aa-b616-4a9e-9dd9-ae7eefd3e847-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd8986aa-b616-4a9e-9dd9-ae7eefd3e847-istio-envoy\") pod \"bd8986aa-b616-4a9e-9dd9-ae7eefd3e847\" (UID: \"bd8986aa-b616-4a9e-9dd9-ae7eefd3e847\") "
Oct 30 22:28:34.945349 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:34.945330    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd8986aa-b616-4a9e-9dd9-ae7eefd3e847-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd8986aa-b616-4a9e-9dd9-ae7eefd3e847-workload-certs\") pod \"bd8986aa-b616-4a9e-9dd9-ae7eefd3e847\" (UID: \"bd8986aa-b616-4a9e-9dd9-ae7eefd3e847\") "
Oct 30 22:28:34.945977 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:34.944812    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd8986aa-b616-4a9e-9dd9-ae7eefd3e847-istio-data" pod "bd8986aa-b616-4a9e-9dd9-ae7eefd3e847" (UID: "bd8986aa-b616-4a9e-9dd9-ae7eefd3e847"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:28:34.945977 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:34.945680    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd8986aa-b616-4a9e-9dd9-ae7eefd3e847-istiod-ca-cert" pod "bd8986aa-b616-4a9e-9dd9-ae7eefd3e847" (UID: "bd8986aa-b616-4a9e-9dd9-ae7eefd3e847"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:28:34.947891 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:34.946494    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd8986aa-b616-4a9e-9dd9-ae7eefd3e847-credential-socket" pod "bd8986aa-b616-4a9e-9dd9-ae7eefd3e847" (UID: "bd8986aa-b616-4a9e-9dd9-ae7eefd3e847"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:28:34.947891 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:34.946910    5678 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd8986aa-b616-4a9e-9dd9-ae7eefd3e847-istiod-ca-cert\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:34.947891 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:34.946935    5678 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd8986aa-b616-4a9e-9dd9-ae7eefd3e847-credential-socket\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:34.947891 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:34.946950    5678 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd8986aa-b616-4a9e-9dd9-ae7eefd3e847-workload-socket\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:34.947891 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:34.946965    5678 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd8986aa-b616-4a9e-9dd9-ae7eefd3e847-istio-data\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:34.947891 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:34.947103    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bd8986aa-b616-4a9e-9dd9-ae7eefd3e847-istio-podinfo" pod "bd8986aa-b616-4a9e-9dd9-ae7eefd3e847" (UID: "bd8986aa-b616-4a9e-9dd9-ae7eefd3e847"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:28:34.947891 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:34.947853    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd8986aa-b616-4a9e-9dd9-ae7eefd3e847-workload-certs" pod "bd8986aa-b616-4a9e-9dd9-ae7eefd3e847" (UID: "bd8986aa-b616-4a9e-9dd9-ae7eefd3e847"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:28:34.948357 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:34.948332    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd8986aa-b616-4a9e-9dd9-ae7eefd3e847-kube-api-access-cp5tl" pod "bd8986aa-b616-4a9e-9dd9-ae7eefd3e847" (UID: "bd8986aa-b616-4a9e-9dd9-ae7eefd3e847"). InnerVolumeSpecName "kube-api-access-cp5tl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:28:34.949571 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:34.949437    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd8986aa-b616-4a9e-9dd9-ae7eefd3e847-istio-envoy" pod "bd8986aa-b616-4a9e-9dd9-ae7eefd3e847" (UID: "bd8986aa-b616-4a9e-9dd9-ae7eefd3e847"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:28:34.949571 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:34.949482    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd8986aa-b616-4a9e-9dd9-ae7eefd3e847-istio-token" pod "bd8986aa-b616-4a9e-9dd9-ae7eefd3e847" (UID: "bd8986aa-b616-4a9e-9dd9-ae7eefd3e847"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:28:35.065750 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:35.065709    5678 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd8986aa-b616-4a9e-9dd9-ae7eefd3e847-istio-podinfo\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:35.065750 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:35.065745    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cp5tl\" (UniqueName: \"kubernetes.io/projected/bd8986aa-b616-4a9e-9dd9-ae7eefd3e847-kube-api-access-cp5tl\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:35.065750 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:35.065761    5678 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd8986aa-b616-4a9e-9dd9-ae7eefd3e847-istio-token\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:35.066154 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:35.065773    5678 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd8986aa-b616-4a9e-9dd9-ae7eefd3e847-istio-envoy\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:35.066154 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:35.065786    5678 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd8986aa-b616-4a9e-9dd9-ae7eefd3e847-workload-certs\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:35.580316 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:35.580289    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c24d61-openshift-default-7c56bd489-c85l8"
Oct 30 22:28:35.580837 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:35.580321    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c24d61-openshift-default-7c56bd489-c85l8" event={"ID":"bd8986aa-b616-4a9e-9dd9-ae7eefd3e847","Type":"ContainerDied","Data":"7aacd52bae060401b4048f72cd22b8fd51bf26192fe905640071e0e0aab3fbe9"}
Oct 30 22:28:35.580837 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:35.580375    5678 scope.go:122] "RemoveContainer" containerID="69ca4fb4a4f0d20bb76fac59352f175389dd3e67b6534374f201b30d9c3999ba"
Oct 30 22:28:35.785748 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:35.785712    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-c24d61-openshift-default-7c56bd489-c85l8"]
Oct 30 22:28:35.833043 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:35.832960    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-c24d61-openshift-default-7c56bd489-c85l8"]
Oct 30 22:28:36.338318 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:36.338284    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-65e59f/distinct-2-w4-65e59f-kserve-689566757d-bf98q"]
Oct 30 22:28:36.363709 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:36.363675    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-65e59f/premium-w4-65e59f-kserve-6dc49468f8-rlzhx"]
Oct 30 22:28:36.373308 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:36.373276    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-65e59f/simulator-w4-65e59f-kserve-85698bcd47-przch"]
Oct 30 22:28:36.547743 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:36.547708    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd8986aa-b616-4a9e-9dd9-ae7eefd3e847" path="/var/lib/kubelet/pods/bd8986aa-b616-4a9e-9dd9-ae7eefd3e847/volumes"
Oct 30 22:28:40.554244 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:40.554211    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d7cdcc-c769d69c6-7z9lp"]
Oct 30 22:28:40.554661 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:40.554472    5678 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d7cdcc-c769d69c6-7z9lp" podUID="1686023a-bf9e-4224-82ea-1c72bc4273c6" containerName="maas-api" containerID="cri-o://45663b3cc455a133e35849a4c7acf727a79c51f365416d0d2bec750b0c18871b" gracePeriod=30
Oct 30 22:28:40.577927 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:28:40.577899    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 22:28:40.679101 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:40.679069    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d7cdcc-c769d69c6-7z9lp" podUID="1686023a-bf9e-4224-82ea-1c72bc4273c6" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.144:8443/health\": dial tcp 10.134.0.144:8443: connect: connection refused"
Oct 30 22:28:40.850243 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:40.850210    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d7cdcc-c769d69c6-7z9lp"
Oct 30 22:28:40.971247 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:40.971204    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1686023a-bf9e-4224-82ea-1c72bc4273c6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1686023a-bf9e-4224-82ea-1c72bc4273c6-maas-api-tls\") pod \"1686023a-bf9e-4224-82ea-1c72bc4273c6\" (UID: \"1686023a-bf9e-4224-82ea-1c72bc4273c6\") "
Oct 30 22:28:40.971473 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:40.971315    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1686023a-bf9e-4224-82ea-1c72bc4273c6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1686023a-bf9e-4224-82ea-1c72bc4273c6-openshift-service-ca\") pod \"1686023a-bf9e-4224-82ea-1c72bc4273c6\" (UID: \"1686023a-bf9e-4224-82ea-1c72bc4273c6\") "
Oct 30 22:28:40.971612 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:40.971586    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1686023a-bf9e-4224-82ea-1c72bc4273c6-kube-api-access-t8qlm\" (UniqueName: \"kubernetes.io/projected/1686023a-bf9e-4224-82ea-1c72bc4273c6-kube-api-access-t8qlm\") pod \"1686023a-bf9e-4224-82ea-1c72bc4273c6\" (UID: \"1686023a-bf9e-4224-82ea-1c72bc4273c6\") "
Oct 30 22:28:40.971748 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:40.971670    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1686023a-bf9e-4224-82ea-1c72bc4273c6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1686023a-bf9e-4224-82ea-1c72bc4273c6-metrics-certs\") pod \"1686023a-bf9e-4224-82ea-1c72bc4273c6\" (UID: \"1686023a-bf9e-4224-82ea-1c72bc4273c6\") "
Oct 30 22:28:40.971860 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:40.971798    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1686023a-bf9e-4224-82ea-1c72bc4273c6-openshift-service-ca" pod "1686023a-bf9e-4224-82ea-1c72bc4273c6" (UID: "1686023a-bf9e-4224-82ea-1c72bc4273c6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:28:40.972903 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:40.972882    5678 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1686023a-bf9e-4224-82ea-1c72bc4273c6-openshift-service-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:40.976921 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:40.976892    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1686023a-bf9e-4224-82ea-1c72bc4273c6-maas-api-tls" pod "1686023a-bf9e-4224-82ea-1c72bc4273c6" (UID: "1686023a-bf9e-4224-82ea-1c72bc4273c6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:28:40.977209 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:40.977158    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1686023a-bf9e-4224-82ea-1c72bc4273c6-metrics-certs" pod "1686023a-bf9e-4224-82ea-1c72bc4273c6" (UID: "1686023a-bf9e-4224-82ea-1c72bc4273c6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:28:40.995906 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:40.995653    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1686023a-bf9e-4224-82ea-1c72bc4273c6-kube-api-access-t8qlm" pod "1686023a-bf9e-4224-82ea-1c72bc4273c6" (UID: "1686023a-bf9e-4224-82ea-1c72bc4273c6"). InnerVolumeSpecName "kube-api-access-t8qlm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:28:41.074530 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:41.074434    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8qlm\" (UniqueName: \"kubernetes.io/projected/1686023a-bf9e-4224-82ea-1c72bc4273c6-kube-api-access-t8qlm\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:41.074530 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:41.074479    5678 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1686023a-bf9e-4224-82ea-1c72bc4273c6-metrics-certs\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:41.074530 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:41.074493    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1686023a-bf9e-4224-82ea-1c72bc4273c6-maas-api-tls\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:41.626718 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:41.626683    5678 generic.go:356] "Generic (PLEG): container finished" podID="1686023a-bf9e-4224-82ea-1c72bc4273c6" containerID="45663b3cc455a133e35849a4c7acf727a79c51f365416d0d2bec750b0c18871b" exitCode=0
Oct 30 22:28:41.626718 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:41.626713    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d7cdcc-c769d69c6-7z9lp" event={"ID":"1686023a-bf9e-4224-82ea-1c72bc4273c6","Type":"ContainerDied","Data":"45663b3cc455a133e35849a4c7acf727a79c51f365416d0d2bec750b0c18871b"}
Oct 30 22:28:41.627257 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:41.626732    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d7cdcc-c769d69c6-7z9lp"
Oct 30 22:28:41.627257 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:41.626766    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d7cdcc-c769d69c6-7z9lp" event={"ID":"1686023a-bf9e-4224-82ea-1c72bc4273c6","Type":"ContainerDied","Data":"06d733c2d2ad6bf3200f7d4ecdcfc1da98a7b18be77e7ab771ee48ab1b4872db"}
Oct 30 22:28:41.627257 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:41.626788    5678 scope.go:122] "RemoveContainer" containerID="45663b3cc455a133e35849a4c7acf727a79c51f365416d0d2bec750b0c18871b"
Oct 30 22:28:41.637975 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:41.637955    5678 scope.go:122] "RemoveContainer" containerID="45663b3cc455a133e35849a4c7acf727a79c51f365416d0d2bec750b0c18871b"
Oct 30 22:28:41.638247 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:28:41.638225    5678 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45663b3cc455a133e35849a4c7acf727a79c51f365416d0d2bec750b0c18871b\": container with ID starting with 45663b3cc455a133e35849a4c7acf727a79c51f365416d0d2bec750b0c18871b not found: ID does not exist" containerID="45663b3cc455a133e35849a4c7acf727a79c51f365416d0d2bec750b0c18871b"
Oct 30 22:28:41.638315 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:41.638260    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45663b3cc455a133e35849a4c7acf727a79c51f365416d0d2bec750b0c18871b"} err="failed to get container status \"45663b3cc455a133e35849a4c7acf727a79c51f365416d0d2bec750b0c18871b\": rpc error: code = NotFound desc = could not find container \"45663b3cc455a133e35849a4c7acf727a79c51f365416d0d2bec750b0c18871b\": container with ID starting with 45663b3cc455a133e35849a4c7acf727a79c51f365416d0d2bec750b0c18871b not found: ID does not exist"
Oct 30 22:28:41.785162 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:41.785128    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d7cdcc-c769d69c6-7z9lp"]
Oct 30 22:28:41.832487 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:41.832393    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d7cdcc-c769d69c6-7z9lp"]
Oct 30 22:28:41.837908 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:41.837758    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0b6a3f-f9776f9d7-rxn5c"]
Oct 30 22:28:41.838107 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:41.838061    5678 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0b6a3f-f9776f9d7-rxn5c" podUID="df2fee05-175f-44c9-9640-b648c8744981" containerName="maas-api" containerID="cri-o://30aff90aab0591f80371a2de554ba3559f08f4833a989bddbde0db1ffda1b062" gracePeriod=30
Oct 30 22:28:42.148459 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:42.148430    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0b6a3f-f9776f9d7-rxn5c"
Oct 30 22:28:42.206477 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:42.206444    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df2fee05-175f-44c9-9640-b648c8744981-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df2fee05-175f-44c9-9640-b648c8744981-maas-api-tls\") pod \"df2fee05-175f-44c9-9640-b648c8744981\" (UID: \"df2fee05-175f-44c9-9640-b648c8744981\") "
Oct 30 22:28:42.206646 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:42.206556    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df2fee05-175f-44c9-9640-b648c8744981-kube-api-access-8hvvg\" (UniqueName: \"kubernetes.io/projected/df2fee05-175f-44c9-9640-b648c8744981-kube-api-access-8hvvg\") pod \"df2fee05-175f-44c9-9640-b648c8744981\" (UID: \"df2fee05-175f-44c9-9640-b648c8744981\") "
Oct 30 22:28:42.206646 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:42.206598    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df2fee05-175f-44c9-9640-b648c8744981-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df2fee05-175f-44c9-9640-b648c8744981-openshift-service-ca\") pod \"df2fee05-175f-44c9-9640-b648c8744981\" (UID: \"df2fee05-175f-44c9-9640-b648c8744981\") "
Oct 30 22:28:42.206738 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:42.206679    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df2fee05-175f-44c9-9640-b648c8744981-metrics-certs\" (UniqueName: \"kubernetes.io/secret/df2fee05-175f-44c9-9640-b648c8744981-metrics-certs\") pod \"df2fee05-175f-44c9-9640-b648c8744981\" (UID: \"df2fee05-175f-44c9-9640-b648c8744981\") "
Oct 30 22:28:42.206987 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:42.206954    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df2fee05-175f-44c9-9640-b648c8744981-openshift-service-ca" pod "df2fee05-175f-44c9-9640-b648c8744981" (UID: "df2fee05-175f-44c9-9640-b648c8744981"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:28:42.207276 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:42.207254    5678 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df2fee05-175f-44c9-9640-b648c8744981-openshift-service-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:42.208762 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:42.208704    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df2fee05-175f-44c9-9640-b648c8744981-kube-api-access-8hvvg" pod "df2fee05-175f-44c9-9640-b648c8744981" (UID: "df2fee05-175f-44c9-9640-b648c8744981"). InnerVolumeSpecName "kube-api-access-8hvvg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:28:42.209110 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:42.209097    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df2fee05-175f-44c9-9640-b648c8744981-maas-api-tls" pod "df2fee05-175f-44c9-9640-b648c8744981" (UID: "df2fee05-175f-44c9-9640-b648c8744981"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:28:42.209176 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:42.209128    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df2fee05-175f-44c9-9640-b648c8744981-metrics-certs" pod "df2fee05-175f-44c9-9640-b648c8744981" (UID: "df2fee05-175f-44c9-9640-b648c8744981"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:28:42.316726 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:42.316684    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hvvg\" (UniqueName: \"kubernetes.io/projected/df2fee05-175f-44c9-9640-b648c8744981-kube-api-access-8hvvg\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:42.316726 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:42.316728    5678 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df2fee05-175f-44c9-9640-b648c8744981-metrics-certs\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:42.316951 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:42.316744    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df2fee05-175f-44c9-9640-b648c8744981-maas-api-tls\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:42.526500 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:42.526412    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1686023a-bf9e-4224-82ea-1c72bc4273c6" path="/var/lib/kubelet/pods/1686023a-bf9e-4224-82ea-1c72bc4273c6/volumes"
Oct 30 22:28:42.642387 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:42.642222    5678 generic.go:356] "Generic (PLEG): container finished" podID="df2fee05-175f-44c9-9640-b648c8744981" containerID="30aff90aab0591f80371a2de554ba3559f08f4833a989bddbde0db1ffda1b062" exitCode=0
Oct 30 22:28:42.643010 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:42.642965    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0b6a3f-f9776f9d7-rxn5c" event={"ID":"df2fee05-175f-44c9-9640-b648c8744981","Type":"ContainerDied","Data":"30aff90aab0591f80371a2de554ba3559f08f4833a989bddbde0db1ffda1b062"}
Oct 30 22:28:42.643126 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:42.643039    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0b6a3f-f9776f9d7-rxn5c" event={"ID":"df2fee05-175f-44c9-9640-b648c8744981","Type":"ContainerDied","Data":"bdbee2d815452ec94b0411880d99d130d8158e89fa9da896042514f129547035"}
Oct 30 22:28:42.643126 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:42.643074    5678 scope.go:122] "RemoveContainer" containerID="30aff90aab0591f80371a2de554ba3559f08f4833a989bddbde0db1ffda1b062"
Oct 30 22:28:42.643254 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:42.643240    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0b6a3f-f9776f9d7-rxn5c"
Oct 30 22:28:42.658715 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:42.658696    5678 scope.go:122] "RemoveContainer" containerID="30aff90aab0591f80371a2de554ba3559f08f4833a989bddbde0db1ffda1b062"
Oct 30 22:28:42.659002 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:28:42.658981    5678 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30aff90aab0591f80371a2de554ba3559f08f4833a989bddbde0db1ffda1b062\": container with ID starting with 30aff90aab0591f80371a2de554ba3559f08f4833a989bddbde0db1ffda1b062 not found: ID does not exist" containerID="30aff90aab0591f80371a2de554ba3559f08f4833a989bddbde0db1ffda1b062"
Oct 30 22:28:42.659075 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:42.659011    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30aff90aab0591f80371a2de554ba3559f08f4833a989bddbde0db1ffda1b062"} err="failed to get container status \"30aff90aab0591f80371a2de554ba3559f08f4833a989bddbde0db1ffda1b062\": rpc error: code = NotFound desc = could not find container \"30aff90aab0591f80371a2de554ba3559f08f4833a989bddbde0db1ffda1b062\": container with ID starting with 30aff90aab0591f80371a2de554ba3559f08f4833a989bddbde0db1ffda1b062 not found: ID does not exist"
Oct 30 22:28:42.808474 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:42.808358    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0b6a3f-f9776f9d7-rxn5c"]
Oct 30 22:28:42.811689 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:42.811660    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0b6a3f-f9776f9d7-rxn5c"]
Oct 30 22:28:44.189968 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:44.189938    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-65e59f-74d8d9b4c-t64kq"]
Oct 30 22:28:44.549974 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:44.549900    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df2fee05-175f-44c9-9640-b648c8744981" path="/var/lib/kubelet/pods/df2fee05-175f-44c9-9640-b648c8744981/volumes"
Oct 30 22:28:49.191230 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:49.191164    5678 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-65e59f-74d8d9b4c-t64kq" podUID="4b088ba1-b952-4a98-9316-7673c67ad3c8" containerName="payload-pre-processing" containerID="cri-o://8d7f8b4b344dbd3851a421b5cabafb83cdc35454617ae8b7c82b8e720979bcda" gracePeriod=25
Oct 30 22:28:50.130999 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:50.130966    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-65e59f-74d8d9b4c-t64kq" podUID="4b088ba1-b952-4a98-9316-7673c67ad3c8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:28:51.359282 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:51.359239    5678 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-65e59f/distinct-2-w4-65e59f-kserve-689566757d-bf98q" podUID="64e66fdb-056e-4086-a85a-8c02613584a5" containerName="main" containerID="cri-o://f3733e4c04c8dc60bd942cdf795c540ecb4b8b621d3f6debc73088508c8880b9" gracePeriod=45
Oct 30 22:28:51.384121 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:51.384084    5678 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-65e59f/premium-w4-65e59f-kserve-6dc49468f8-rlzhx" podUID="8590a783-eb24-4759-ad74-1369f36aa9d2" containerName="main" containerID="cri-o://0e4df1de02bb062c21465c899d86aabb41fe51a6cef551344991550623680d7f" gracePeriod=45
Oct 30 22:28:51.393767 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:51.393729    5678 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-65e59f/simulator-w4-65e59f-kserve-85698bcd47-przch" podUID="ee5e843b-17be-4623-8c95-dc07c8b7f506" containerName="main" containerID="cri-o://bea563ed6f19fd1ea6443895748dd6fd1c575a1cac8de515cfde7bb974e3c2dd" gracePeriod=45
Oct 30 22:28:51.491361 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:51.491317    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-65e59f/distinct-2-w4-65e59f-kserve-689566757d-bf98q" podUID="64e66fdb-056e-4086-a85a-8c02613584a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": dial tcp 10.134.0.92:8000: connect: connection refused"
Oct 30 22:28:51.707269 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:51.707242    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-65e59f/distinct-2-w4-65e59f-kserve-689566757d-bf98q"
Oct 30 22:28:51.715018 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:51.714990    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-65e59f/simulator-w4-65e59f-kserve-85698bcd47-przch"
Oct 30 22:28:51.718891 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:51.718870    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-65e59f/premium-w4-65e59f-kserve-6dc49468f8-rlzhx"
Oct 30 22:28:51.719273 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:51.719248    5678 generic.go:356] "Generic (PLEG): container finished" podID="64e66fdb-056e-4086-a85a-8c02613584a5" containerID="f3733e4c04c8dc60bd942cdf795c540ecb4b8b621d3f6debc73088508c8880b9" exitCode=0
Oct 30 22:28:51.719499 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:51.719479    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-65e59f/distinct-2-w4-65e59f-kserve-689566757d-bf98q"
Oct 30 22:28:51.719499 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:51.719489    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-65e59f/distinct-2-w4-65e59f-kserve-689566757d-bf98q" event={"ID":"64e66fdb-056e-4086-a85a-8c02613584a5","Type":"ContainerDied","Data":"f3733e4c04c8dc60bd942cdf795c540ecb4b8b621d3f6debc73088508c8880b9"}
Oct 30 22:28:51.719594 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:51.719530    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-65e59f/distinct-2-w4-65e59f-kserve-689566757d-bf98q" event={"ID":"64e66fdb-056e-4086-a85a-8c02613584a5","Type":"ContainerDied","Data":"da2545beef0e2cb33fe413da44cc8ad4fd599e875f3e77d35fa2cd47b0cb9cb4"}
Oct 30 22:28:51.719594 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:51.719553    5678 scope.go:122] "RemoveContainer" containerID="f3733e4c04c8dc60bd942cdf795c540ecb4b8b621d3f6debc73088508c8880b9"
Oct 30 22:28:51.721013 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:51.720984    5678 generic.go:356] "Generic (PLEG): container finished" podID="ee5e843b-17be-4623-8c95-dc07c8b7f506" containerID="bea563ed6f19fd1ea6443895748dd6fd1c575a1cac8de515cfde7bb974e3c2dd" exitCode=0
Oct 30 22:28:51.721109 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:51.721022    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-65e59f/simulator-w4-65e59f-kserve-85698bcd47-przch"
Oct 30 22:28:51.721109 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:51.721028    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-65e59f/simulator-w4-65e59f-kserve-85698bcd47-przch" event={"ID":"ee5e843b-17be-4623-8c95-dc07c8b7f506","Type":"ContainerDied","Data":"bea563ed6f19fd1ea6443895748dd6fd1c575a1cac8de515cfde7bb974e3c2dd"}
Oct 30 22:28:51.721109 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:51.721057    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-65e59f/simulator-w4-65e59f-kserve-85698bcd47-przch" event={"ID":"ee5e843b-17be-4623-8c95-dc07c8b7f506","Type":"ContainerDied","Data":"e8560772cd00e0f348be6afbf5fa84c4dac449d462615c38bb0aa966e61aea65"}
Oct 30 22:28:51.722490 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:51.722465    5678 generic.go:356] "Generic (PLEG): container finished" podID="8590a783-eb24-4759-ad74-1369f36aa9d2" containerID="0e4df1de02bb062c21465c899d86aabb41fe51a6cef551344991550623680d7f" exitCode=0
Oct 30 22:28:51.722566 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:51.722504    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-65e59f/premium-w4-65e59f-kserve-6dc49468f8-rlzhx" event={"ID":"8590a783-eb24-4759-ad74-1369f36aa9d2","Type":"ContainerDied","Data":"0e4df1de02bb062c21465c899d86aabb41fe51a6cef551344991550623680d7f"}
Oct 30 22:28:51.722566 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:51.722510    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-65e59f/premium-w4-65e59f-kserve-6dc49468f8-rlzhx"
Oct 30 22:28:51.722566 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:51.722530    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-65e59f/premium-w4-65e59f-kserve-6dc49468f8-rlzhx" event={"ID":"8590a783-eb24-4759-ad74-1369f36aa9d2","Type":"ContainerDied","Data":"26a0b42d67819a9271e7adaf0b58b3150bcedb414e430aebfcd9c59c44fc9b1d"}
Oct 30 22:28:51.730029 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:51.729952    5678 scope.go:122] "RemoveContainer" containerID="f3733e4c04c8dc60bd942cdf795c540ecb4b8b621d3f6debc73088508c8880b9"
Oct 30 22:28:51.732819 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:28:51.732789    5678 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3733e4c04c8dc60bd942cdf795c540ecb4b8b621d3f6debc73088508c8880b9\": container with ID starting with f3733e4c04c8dc60bd942cdf795c540ecb4b8b621d3f6debc73088508c8880b9 not found: ID does not exist" containerID="f3733e4c04c8dc60bd942cdf795c540ecb4b8b621d3f6debc73088508c8880b9"
Oct 30 22:28:51.732922 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:51.732829    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3733e4c04c8dc60bd942cdf795c540ecb4b8b621d3f6debc73088508c8880b9"} err="failed to get container status \"f3733e4c04c8dc60bd942cdf795c540ecb4b8b621d3f6debc73088508c8880b9\": rpc error: code = NotFound desc = could not find container \"f3733e4c04c8dc60bd942cdf795c540ecb4b8b621d3f6debc73088508c8880b9\": container with ID starting with f3733e4c04c8dc60bd942cdf795c540ecb4b8b621d3f6debc73088508c8880b9 not found: ID does not exist"
Oct 30 22:28:51.732922 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:51.732850    5678 scope.go:122] "RemoveContainer" containerID="bea563ed6f19fd1ea6443895748dd6fd1c575a1cac8de515cfde7bb974e3c2dd"
Oct 30 22:28:51.741966 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:51.741949    5678 scope.go:122] "RemoveContainer" containerID="bea563ed6f19fd1ea6443895748dd6fd1c575a1cac8de515cfde7bb974e3c2dd"
Oct 30 22:28:51.742217 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:28:51.742201    5678 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bea563ed6f19fd1ea6443895748dd6fd1c575a1cac8de515cfde7bb974e3c2dd\": container with ID starting with bea563ed6f19fd1ea6443895748dd6fd1c575a1cac8de515cfde7bb974e3c2dd not found: ID does not exist" containerID="bea563ed6f19fd1ea6443895748dd6fd1c575a1cac8de515cfde7bb974e3c2dd"
Oct 30 22:28:51.742264 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:51.742223    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bea563ed6f19fd1ea6443895748dd6fd1c575a1cac8de515cfde7bb974e3c2dd"} err="failed to get container status \"bea563ed6f19fd1ea6443895748dd6fd1c575a1cac8de515cfde7bb974e3c2dd\": rpc error: code = NotFound desc = could not find container \"bea563ed6f19fd1ea6443895748dd6fd1c575a1cac8de515cfde7bb974e3c2dd\": container with ID starting with bea563ed6f19fd1ea6443895748dd6fd1c575a1cac8de515cfde7bb974e3c2dd not found: ID does not exist"
Oct 30 22:28:51.742264 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:51.742238    5678 scope.go:122] "RemoveContainer" containerID="0e4df1de02bb062c21465c899d86aabb41fe51a6cef551344991550623680d7f"
Oct 30 22:28:51.751170 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:51.751153    5678 scope.go:122] "RemoveContainer" containerID="0e4df1de02bb062c21465c899d86aabb41fe51a6cef551344991550623680d7f"
Oct 30 22:28:51.751483 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:28:51.751463    5678 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e4df1de02bb062c21465c899d86aabb41fe51a6cef551344991550623680d7f\": container with ID starting with 0e4df1de02bb062c21465c899d86aabb41fe51a6cef551344991550623680d7f not found: ID does not exist" containerID="0e4df1de02bb062c21465c899d86aabb41fe51a6cef551344991550623680d7f"
Oct 30 22:28:51.751553 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:51.751493    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e4df1de02bb062c21465c899d86aabb41fe51a6cef551344991550623680d7f"} err="failed to get container status \"0e4df1de02bb062c21465c899d86aabb41fe51a6cef551344991550623680d7f\": rpc error: code = NotFound desc = could not find container \"0e4df1de02bb062c21465c899d86aabb41fe51a6cef551344991550623680d7f\": container with ID starting with 0e4df1de02bb062c21465c899d86aabb41fe51a6cef551344991550623680d7f not found: ID does not exist"
Oct 30 22:28:51.763527 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:51.763505    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64e66fdb-056e-4086-a85a-8c02613584a5-home\" (UniqueName: \"kubernetes.io/empty-dir/64e66fdb-056e-4086-a85a-8c02613584a5-home\") pod \"64e66fdb-056e-4086-a85a-8c02613584a5\" (UID: \"64e66fdb-056e-4086-a85a-8c02613584a5\") "
Oct 30 22:28:51.763650 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:51.763562    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64e66fdb-056e-4086-a85a-8c02613584a5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64e66fdb-056e-4086-a85a-8c02613584a5-tmp-dir\") pod \"64e66fdb-056e-4086-a85a-8c02613584a5\" (UID: \"64e66fdb-056e-4086-a85a-8c02613584a5\") "
Oct 30 22:28:51.763650 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:51.763624    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64e66fdb-056e-4086-a85a-8c02613584a5-kube-api-access-np7vq\" (UniqueName: \"kubernetes.io/projected/64e66fdb-056e-4086-a85a-8c02613584a5-kube-api-access-np7vq\") pod \"64e66fdb-056e-4086-a85a-8c02613584a5\" (UID: \"64e66fdb-056e-4086-a85a-8c02613584a5\") "
Oct 30 22:28:51.763787 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:51.763771    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64e66fdb-056e-4086-a85a-8c02613584a5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/64e66fdb-056e-4086-a85a-8c02613584a5-dshm\") pod \"64e66fdb-056e-4086-a85a-8c02613584a5\" (UID: \"64e66fdb-056e-4086-a85a-8c02613584a5\") "
Oct 30 22:28:51.763846 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:51.763791    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64e66fdb-056e-4086-a85a-8c02613584a5-tmp-dir" pod "64e66fdb-056e-4086-a85a-8c02613584a5" (UID: "64e66fdb-056e-4086-a85a-8c02613584a5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:28:51.763846 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:51.763805    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64e66fdb-056e-4086-a85a-8c02613584a5-home" pod "64e66fdb-056e-4086-a85a-8c02613584a5" (UID: "64e66fdb-056e-4086-a85a-8c02613584a5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:28:51.763846 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:51.763830    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64e66fdb-056e-4086-a85a-8c02613584a5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64e66fdb-056e-4086-a85a-8c02613584a5-model-cache\") pod \"64e66fdb-056e-4086-a85a-8c02613584a5\" (UID: \"64e66fdb-056e-4086-a85a-8c02613584a5\") "
Oct 30 22:28:51.763999 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:51.763949    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64e66fdb-056e-4086-a85a-8c02613584a5-tls-certs\" (UniqueName: \"kubernetes.io/secret/64e66fdb-056e-4086-a85a-8c02613584a5-tls-certs\") pod \"64e66fdb-056e-4086-a85a-8c02613584a5\" (UID: \"64e66fdb-056e-4086-a85a-8c02613584a5\") "
Oct 30 22:28:51.764129 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:51.764099    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64e66fdb-056e-4086-a85a-8c02613584a5-model-cache" pod "64e66fdb-056e-4086-a85a-8c02613584a5" (UID: "64e66fdb-056e-4086-a85a-8c02613584a5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:28:51.764516 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:51.764497    5678 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64e66fdb-056e-4086-a85a-8c02613584a5-model-cache\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:51.764627 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:51.764519    5678 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64e66fdb-056e-4086-a85a-8c02613584a5-home\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:51.764627 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:51.764529    5678 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64e66fdb-056e-4086-a85a-8c02613584a5-tmp-dir\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:51.765863 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:51.765836    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64e66fdb-056e-4086-a85a-8c02613584a5-kube-api-access-np7vq" pod "64e66fdb-056e-4086-a85a-8c02613584a5" (UID: "64e66fdb-056e-4086-a85a-8c02613584a5"). InnerVolumeSpecName "kube-api-access-np7vq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:28:51.765947 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:51.765886    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64e66fdb-056e-4086-a85a-8c02613584a5-dshm" pod "64e66fdb-056e-4086-a85a-8c02613584a5" (UID: "64e66fdb-056e-4086-a85a-8c02613584a5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:28:51.765947 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:51.765930    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64e66fdb-056e-4086-a85a-8c02613584a5-tls-certs" pod "64e66fdb-056e-4086-a85a-8c02613584a5" (UID: "64e66fdb-056e-4086-a85a-8c02613584a5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:28:51.865755 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:51.865716    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee5e843b-17be-4623-8c95-dc07c8b7f506-home\" (UniqueName: \"kubernetes.io/empty-dir/ee5e843b-17be-4623-8c95-dc07c8b7f506-home\") pod \"ee5e843b-17be-4623-8c95-dc07c8b7f506\" (UID: \"ee5e843b-17be-4623-8c95-dc07c8b7f506\") "
Oct 30 22:28:51.865944 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:51.865869    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8590a783-eb24-4759-ad74-1369f36aa9d2-kube-api-access-6xfxz\" (UniqueName: \"kubernetes.io/projected/8590a783-eb24-4759-ad74-1369f36aa9d2-kube-api-access-6xfxz\") pod \"8590a783-eb24-4759-ad74-1369f36aa9d2\" (UID: \"8590a783-eb24-4759-ad74-1369f36aa9d2\") "
Oct 30 22:28:51.865944 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:51.865916    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8590a783-eb24-4759-ad74-1369f36aa9d2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8590a783-eb24-4759-ad74-1369f36aa9d2-tmp-dir\") pod \"8590a783-eb24-4759-ad74-1369f36aa9d2\" (UID: \"8590a783-eb24-4759-ad74-1369f36aa9d2\") "
Oct 30 22:28:51.866081 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:51.865941    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8590a783-eb24-4759-ad74-1369f36aa9d2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8590a783-eb24-4759-ad74-1369f36aa9d2-dshm\") pod \"8590a783-eb24-4759-ad74-1369f36aa9d2\" (UID: \"8590a783-eb24-4759-ad74-1369f36aa9d2\") "
Oct 30 22:28:51.866081 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:51.866031    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee5e843b-17be-4623-8c95-dc07c8b7f506-home" pod "ee5e843b-17be-4623-8c95-dc07c8b7f506" (UID: "ee5e843b-17be-4623-8c95-dc07c8b7f506"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:28:51.866206 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:51.866149    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8590a783-eb24-4759-ad74-1369f36aa9d2-home\" (UniqueName: \"kubernetes.io/empty-dir/8590a783-eb24-4759-ad74-1369f36aa9d2-home\") pod \"8590a783-eb24-4759-ad74-1369f36aa9d2\" (UID: \"8590a783-eb24-4759-ad74-1369f36aa9d2\") "
Oct 30 22:28:51.866206 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:51.866188    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee5e843b-17be-4623-8c95-dc07c8b7f506-tls-certs\" (UniqueName: \"kubernetes.io/secret/ee5e843b-17be-4623-8c95-dc07c8b7f506-tls-certs\") pod \"ee5e843b-17be-4623-8c95-dc07c8b7f506\" (UID: \"ee5e843b-17be-4623-8c95-dc07c8b7f506\") "
Oct 30 22:28:51.866323 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:51.866220    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee5e843b-17be-4623-8c95-dc07c8b7f506-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee5e843b-17be-4623-8c95-dc07c8b7f506-model-cache\") pod \"ee5e843b-17be-4623-8c95-dc07c8b7f506\" (UID: \"ee5e843b-17be-4623-8c95-dc07c8b7f506\") "
Oct 30 22:28:51.866323 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:51.866251    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee5e843b-17be-4623-8c95-dc07c8b7f506-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee5e843b-17be-4623-8c95-dc07c8b7f506-tmp-dir\") pod \"ee5e843b-17be-4623-8c95-dc07c8b7f506\" (UID: \"ee5e843b-17be-4623-8c95-dc07c8b7f506\") "
Oct 30 22:28:51.866461 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:51.866335    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee5e843b-17be-4623-8c95-dc07c8b7f506-kube-api-access-fqq92\" (UniqueName: \"kubernetes.io/projected/ee5e843b-17be-4623-8c95-dc07c8b7f506-kube-api-access-fqq92\") pod \"ee5e843b-17be-4623-8c95-dc07c8b7f506\" (UID: \"ee5e843b-17be-4623-8c95-dc07c8b7f506\") "
Oct 30 22:28:51.866461 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:51.866445    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8590a783-eb24-4759-ad74-1369f36aa9d2-tls-certs\" (UniqueName: \"kubernetes.io/secret/8590a783-eb24-4759-ad74-1369f36aa9d2-tls-certs\") pod \"8590a783-eb24-4759-ad74-1369f36aa9d2\" (UID: \"8590a783-eb24-4759-ad74-1369f36aa9d2\") "
Oct 30 22:28:51.866576 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:51.866516    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee5e843b-17be-4623-8c95-dc07c8b7f506-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee5e843b-17be-4623-8c95-dc07c8b7f506-dshm\") pod \"ee5e843b-17be-4623-8c95-dc07c8b7f506\" (UID: \"ee5e843b-17be-4623-8c95-dc07c8b7f506\") "
Oct 30 22:28:51.866576 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:51.866517    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8590a783-eb24-4759-ad74-1369f36aa9d2-tmp-dir" pod "8590a783-eb24-4759-ad74-1369f36aa9d2" (UID: "8590a783-eb24-4759-ad74-1369f36aa9d2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:28:51.866576 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:51.866544    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8590a783-eb24-4759-ad74-1369f36aa9d2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8590a783-eb24-4759-ad74-1369f36aa9d2-model-cache\") pod \"8590a783-eb24-4759-ad74-1369f36aa9d2\" (UID: \"8590a783-eb24-4759-ad74-1369f36aa9d2\") "
Oct 30 22:28:51.866729 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:51.866713    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee5e843b-17be-4623-8c95-dc07c8b7f506-tmp-dir" pod "ee5e843b-17be-4623-8c95-dc07c8b7f506" (UID: "ee5e843b-17be-4623-8c95-dc07c8b7f506"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:28:51.866907 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:51.866887    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8590a783-eb24-4759-ad74-1369f36aa9d2-home" pod "8590a783-eb24-4759-ad74-1369f36aa9d2" (UID: "8590a783-eb24-4759-ad74-1369f36aa9d2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:28:51.867119 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:51.867097    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8590a783-eb24-4759-ad74-1369f36aa9d2-model-cache" pod "8590a783-eb24-4759-ad74-1369f36aa9d2" (UID: "8590a783-eb24-4759-ad74-1369f36aa9d2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:28:51.867317 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:51.867295    5678 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8590a783-eb24-4759-ad74-1369f36aa9d2-tmp-dir\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:51.867395 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:51.867363    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-np7vq\" (UniqueName: \"kubernetes.io/projected/64e66fdb-056e-4086-a85a-8c02613584a5-kube-api-access-np7vq\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:51.867395 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:51.867383    5678 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8590a783-eb24-4759-ad74-1369f36aa9d2-home\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:51.867534 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:51.867411    5678 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee5e843b-17be-4623-8c95-dc07c8b7f506-tmp-dir\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:51.867534 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:51.867423    5678 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64e66fdb-056e-4086-a85a-8c02613584a5-dshm\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:51.867534 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:51.867434    5678 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8590a783-eb24-4759-ad74-1369f36aa9d2-model-cache\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:51.867534 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:51.867448    5678 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee5e843b-17be-4623-8c95-dc07c8b7f506-home\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:51.867534 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:51.867462    5678 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64e66fdb-056e-4086-a85a-8c02613584a5-tls-certs\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:51.868316 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:51.868287    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee5e843b-17be-4623-8c95-dc07c8b7f506-model-cache" pod "ee5e843b-17be-4623-8c95-dc07c8b7f506" (UID: "ee5e843b-17be-4623-8c95-dc07c8b7f506"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:28:51.870376 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:51.869969    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8590a783-eb24-4759-ad74-1369f36aa9d2-dshm" pod "8590a783-eb24-4759-ad74-1369f36aa9d2" (UID: "8590a783-eb24-4759-ad74-1369f36aa9d2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:28:51.870376 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:51.870212    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee5e843b-17be-4623-8c95-dc07c8b7f506-dshm" pod "ee5e843b-17be-4623-8c95-dc07c8b7f506" (UID: "ee5e843b-17be-4623-8c95-dc07c8b7f506"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:28:51.870738 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:51.870522    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee5e843b-17be-4623-8c95-dc07c8b7f506-kube-api-access-fqq92" pod "ee5e843b-17be-4623-8c95-dc07c8b7f506" (UID: "ee5e843b-17be-4623-8c95-dc07c8b7f506"). InnerVolumeSpecName "kube-api-access-fqq92". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:28:51.871734 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:51.871697    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8590a783-eb24-4759-ad74-1369f36aa9d2-kube-api-access-6xfxz" pod "8590a783-eb24-4759-ad74-1369f36aa9d2" (UID: "8590a783-eb24-4759-ad74-1369f36aa9d2"). InnerVolumeSpecName "kube-api-access-6xfxz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:28:51.882441 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:51.881565    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee5e843b-17be-4623-8c95-dc07c8b7f506-tls-certs" pod "ee5e843b-17be-4623-8c95-dc07c8b7f506" (UID: "ee5e843b-17be-4623-8c95-dc07c8b7f506"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:28:51.882441 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:51.881755    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8590a783-eb24-4759-ad74-1369f36aa9d2-tls-certs" pod "8590a783-eb24-4759-ad74-1369f36aa9d2" (UID: "8590a783-eb24-4759-ad74-1369f36aa9d2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:28:51.969160 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:51.969086    5678 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8590a783-eb24-4759-ad74-1369f36aa9d2-tls-certs\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:51.969160 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:51.969116    5678 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee5e843b-17be-4623-8c95-dc07c8b7f506-dshm\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:51.969160 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:51.969126    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xfxz\" (UniqueName: \"kubernetes.io/projected/8590a783-eb24-4759-ad74-1369f36aa9d2-kube-api-access-6xfxz\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:51.969160 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:51.969135    5678 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8590a783-eb24-4759-ad74-1369f36aa9d2-dshm\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:51.969160 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:51.969144    5678 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee5e843b-17be-4623-8c95-dc07c8b7f506-tls-certs\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:51.969160 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:51.969151    5678 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee5e843b-17be-4623-8c95-dc07c8b7f506-model-cache\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:51.969160 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:51.969160    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqq92\" (UniqueName: \"kubernetes.io/projected/ee5e843b-17be-4623-8c95-dc07c8b7f506-kube-api-access-fqq92\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:52.277951 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:52.277854    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-65e59f/premium-w4-65e59f-kserve-6dc49468f8-rlzhx"]
Oct 30 22:28:52.303426 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:52.292827    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-65e59f/premium-w4-65e59f-kserve-6dc49468f8-rlzhx"]
Oct 30 22:28:52.386576 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:52.386546    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-65e59f/simulator-w4-65e59f-kserve-85698bcd47-przch"]
Oct 30 22:28:52.390002 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:52.389971    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-65e59f/simulator-w4-65e59f-kserve-85698bcd47-przch"]
Oct 30 22:28:52.497471 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:52.497444    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-65e59f/distinct-2-w4-65e59f-kserve-689566757d-bf98q"]
Oct 30 22:28:52.501825 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:52.501801    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-65e59f/distinct-2-w4-65e59f-kserve-689566757d-bf98q"]
Oct 30 22:28:52.524662 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:52.524637    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64e66fdb-056e-4086-a85a-8c02613584a5" path="/var/lib/kubelet/pods/64e66fdb-056e-4086-a85a-8c02613584a5/volumes"
Oct 30 22:28:52.525044 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:52.525031    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8590a783-eb24-4759-ad74-1369f36aa9d2" path="/var/lib/kubelet/pods/8590a783-eb24-4759-ad74-1369f36aa9d2/volumes"
Oct 30 22:28:52.525427 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:52.525414    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee5e843b-17be-4623-8c95-dc07c8b7f506" path="/var/lib/kubelet/pods/ee5e843b-17be-4623-8c95-dc07c8b7f506/volumes"
Oct 30 22:28:55.036732 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:55.036702    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-75fb62-openshift-default-c9cc6b459-4f2fg"]
Oct 30 22:28:55.037298 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:55.037284    5678 cpu_manager.go:414] "containerMap: removing container" podUID="ee5e843b-17be-4623-8c95-dc07c8b7f506" containerName="main"
Oct 30 22:28:55.037298 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:55.037299    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee5e843b-17be-4623-8c95-dc07c8b7f506" containerName="main"
Oct 30 22:28:55.037391 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:55.037319    5678 cpu_manager.go:414] "containerMap: removing container" podUID="bd8986aa-b616-4a9e-9dd9-ae7eefd3e847" containerName="istio-proxy"
Oct 30 22:28:55.037391 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:55.037325    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd8986aa-b616-4a9e-9dd9-ae7eefd3e847" containerName="istio-proxy"
Oct 30 22:28:55.037391 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:55.037340    5678 cpu_manager.go:414] "containerMap: removing container" podUID="64e66fdb-056e-4086-a85a-8c02613584a5" containerName="main"
Oct 30 22:28:55.037391 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:55.037348    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64e66fdb-056e-4086-a85a-8c02613584a5" containerName="main"
Oct 30 22:28:55.037391 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:55.037361    5678 cpu_manager.go:414] "containerMap: removing container" podUID="1fce9daf-68c9-41d0-825b-2012bb93c8ba" containerName="main"
Oct 30 22:28:55.037391 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:55.037367    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fce9daf-68c9-41d0-825b-2012bb93c8ba" containerName="main"
Oct 30 22:28:55.037391 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:55.037377    5678 cpu_manager.go:414] "containerMap: removing container" podUID="8590a783-eb24-4759-ad74-1369f36aa9d2" containerName="main"
Oct 30 22:28:55.037391 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:55.037384    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8590a783-eb24-4759-ad74-1369f36aa9d2" containerName="main"
Oct 30 22:28:55.037666 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:55.037428    5678 cpu_manager.go:414] "containerMap: removing container" podUID="1686023a-bf9e-4224-82ea-1c72bc4273c6" containerName="maas-api"
Oct 30 22:28:55.037666 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:55.037438    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1686023a-bf9e-4224-82ea-1c72bc4273c6" containerName="maas-api"
Oct 30 22:28:55.037666 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:55.037454    5678 cpu_manager.go:414] "containerMap: removing container" podUID="9c5d0357-a8f0-484c-9584-4b0184fd5284" containerName="istio-proxy"
Oct 30 22:28:55.037666 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:55.037460    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c5d0357-a8f0-484c-9584-4b0184fd5284" containerName="istio-proxy"
Oct 30 22:28:55.037666 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:55.037465    5678 cpu_manager.go:414] "containerMap: removing container" podUID="df2fee05-175f-44c9-9640-b648c8744981" containerName="maas-api"
Oct 30 22:28:55.037666 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:55.037470    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df2fee05-175f-44c9-9640-b648c8744981" containerName="maas-api"
Oct 30 22:28:55.037666 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:55.037481    5678 cpu_manager.go:414] "containerMap: removing container" podUID="36f0a6a2-54be-49bf-9202-502960cbb134" containerName="main"
Oct 30 22:28:55.037666 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:55.037485    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36f0a6a2-54be-49bf-9202-502960cbb134" containerName="main"
Oct 30 22:28:55.037666 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:55.037497    5678 cpu_manager.go:414] "containerMap: removing container" podUID="a58c911f-e2eb-4820-a6b6-24afe78b5fd4" containerName="main"
Oct 30 22:28:55.037666 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:55.037502    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a58c911f-e2eb-4820-a6b6-24afe78b5fd4" containerName="main"
Oct 30 22:28:55.037666 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:55.037511    5678 cpu_manager.go:414] "containerMap: removing container" podUID="bd8986aa-b616-4a9e-9dd9-ae7eefd3e847" containerName="istio-proxy"
Oct 30 22:28:55.037666 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:55.037540    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd8986aa-b616-4a9e-9dd9-ae7eefd3e847" containerName="istio-proxy"
Oct 30 22:28:55.038005 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:55.037694    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee5e843b-17be-4623-8c95-dc07c8b7f506" containerName="main"
Oct 30 22:28:55.038005 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:55.037707    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd8986aa-b616-4a9e-9dd9-ae7eefd3e847" containerName="istio-proxy"
Oct 30 22:28:55.038005 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:55.037714    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="64e66fdb-056e-4086-a85a-8c02613584a5" containerName="main"
Oct 30 22:28:55.038005 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:55.037722    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fce9daf-68c9-41d0-825b-2012bb93c8ba" containerName="main"
Oct 30 22:28:55.038005 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:55.037728    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="8590a783-eb24-4759-ad74-1369f36aa9d2" containerName="main"
Oct 30 22:28:55.038005 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:55.037735    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="1686023a-bf9e-4224-82ea-1c72bc4273c6" containerName="maas-api"
Oct 30 22:28:55.038005 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:55.037742    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c5d0357-a8f0-484c-9584-4b0184fd5284" containerName="istio-proxy"
Oct 30 22:28:55.038005 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:55.037748    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="df2fee05-175f-44c9-9640-b648c8744981" containerName="maas-api"
Oct 30 22:28:55.038005 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:55.037752    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c5d0357-a8f0-484c-9584-4b0184fd5284" containerName="istio-proxy"
Oct 30 22:28:55.038005 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:55.037758    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="36f0a6a2-54be-49bf-9202-502960cbb134" containerName="main"
Oct 30 22:28:55.038005 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:55.037764    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="a58c911f-e2eb-4820-a6b6-24afe78b5fd4" containerName="main"
Oct 30 22:28:55.038005 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:55.037770    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd8986aa-b616-4a9e-9dd9-ae7eefd3e847" containerName="istio-proxy"
Oct 30 22:28:55.043051 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:55.043033    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-75fb62-openshift-default-c9cc6b459-4f2fg"
Oct 30 22:28:55.093590 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:55.093563    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-75fb62-openshift-default-dockercfg-6slql\""
Oct 30 22:28:55.144948 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:55.144914    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-75fb62-openshift-default-c9cc6b459-4f2fg"]
Oct 30 22:28:55.145248 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:55.145222    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1691924-5354-4690-8709-6a20bfd81a7e-workload-certs\") pod \"e2e-inf-a-75fb62-openshift-default-c9cc6b459-4f2fg\" (UID: \"f1691924-5354-4690-8709-6a20bfd81a7e\") " pod="openshift-ingress/e2e-inf-a-75fb62-openshift-default-c9cc6b459-4f2fg"
Oct 30 22:28:55.145382 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:55.145254    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1691924-5354-4690-8709-6a20bfd81a7e-credential-socket\") pod \"e2e-inf-a-75fb62-openshift-default-c9cc6b459-4f2fg\" (UID: \"f1691924-5354-4690-8709-6a20bfd81a7e\") " pod="openshift-ingress/e2e-inf-a-75fb62-openshift-default-c9cc6b459-4f2fg"
Oct 30 22:28:55.145382 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:55.145331    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqwgx\" (UniqueName: \"kubernetes.io/projected/f1691924-5354-4690-8709-6a20bfd81a7e-kube-api-access-wqwgx\") pod \"e2e-inf-a-75fb62-openshift-default-c9cc6b459-4f2fg\" (UID: \"f1691924-5354-4690-8709-6a20bfd81a7e\") " pod="openshift-ingress/e2e-inf-a-75fb62-openshift-default-c9cc6b459-4f2fg"
Oct 30 22:28:55.145538 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:55.145479    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1691924-5354-4690-8709-6a20bfd81a7e-istiod-ca-cert\") pod \"e2e-inf-a-75fb62-openshift-default-c9cc6b459-4f2fg\" (UID: \"f1691924-5354-4690-8709-6a20bfd81a7e\") " pod="openshift-ingress/e2e-inf-a-75fb62-openshift-default-c9cc6b459-4f2fg"
Oct 30 22:28:55.145538 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:55.145530    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1691924-5354-4690-8709-6a20bfd81a7e-istio-token\") pod \"e2e-inf-a-75fb62-openshift-default-c9cc6b459-4f2fg\" (UID: \"f1691924-5354-4690-8709-6a20bfd81a7e\") " pod="openshift-ingress/e2e-inf-a-75fb62-openshift-default-c9cc6b459-4f2fg"
Oct 30 22:28:55.145660 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:55.145566    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1691924-5354-4690-8709-6a20bfd81a7e-istio-podinfo\") pod \"e2e-inf-a-75fb62-openshift-default-c9cc6b459-4f2fg\" (UID: \"f1691924-5354-4690-8709-6a20bfd81a7e\") " pod="openshift-ingress/e2e-inf-a-75fb62-openshift-default-c9cc6b459-4f2fg"
Oct 30 22:28:55.145737 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:55.145696    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1691924-5354-4690-8709-6a20bfd81a7e-workload-socket\") pod \"e2e-inf-a-75fb62-openshift-default-c9cc6b459-4f2fg\" (UID: \"f1691924-5354-4690-8709-6a20bfd81a7e\") " pod="openshift-ingress/e2e-inf-a-75fb62-openshift-default-c9cc6b459-4f2fg"
Oct 30 22:28:55.145796 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:55.145772    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1691924-5354-4690-8709-6a20bfd81a7e-istio-envoy\") pod \"e2e-inf-a-75fb62-openshift-default-c9cc6b459-4f2fg\" (UID: \"f1691924-5354-4690-8709-6a20bfd81a7e\") " pod="openshift-ingress/e2e-inf-a-75fb62-openshift-default-c9cc6b459-4f2fg"
Oct 30 22:28:55.145912 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:55.145899    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1691924-5354-4690-8709-6a20bfd81a7e-istio-data\") pod \"e2e-inf-a-75fb62-openshift-default-c9cc6b459-4f2fg\" (UID: \"f1691924-5354-4690-8709-6a20bfd81a7e\") " pod="openshift-ingress/e2e-inf-a-75fb62-openshift-default-c9cc6b459-4f2fg"
Oct 30 22:28:55.247129 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:55.247097    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1691924-5354-4690-8709-6a20bfd81a7e-istio-envoy\") pod \"e2e-inf-a-75fb62-openshift-default-c9cc6b459-4f2fg\" (UID: \"f1691924-5354-4690-8709-6a20bfd81a7e\") " pod="openshift-ingress/e2e-inf-a-75fb62-openshift-default-c9cc6b459-4f2fg"
Oct 30 22:28:55.247331 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:55.247183    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1691924-5354-4690-8709-6a20bfd81a7e-istio-data\") pod \"e2e-inf-a-75fb62-openshift-default-c9cc6b459-4f2fg\" (UID: \"f1691924-5354-4690-8709-6a20bfd81a7e\") " pod="openshift-ingress/e2e-inf-a-75fb62-openshift-default-c9cc6b459-4f2fg"
Oct 30 22:28:55.247331 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:55.247233    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1691924-5354-4690-8709-6a20bfd81a7e-workload-certs\") pod \"e2e-inf-a-75fb62-openshift-default-c9cc6b459-4f2fg\" (UID: \"f1691924-5354-4690-8709-6a20bfd81a7e\") " pod="openshift-ingress/e2e-inf-a-75fb62-openshift-default-c9cc6b459-4f2fg"
Oct 30 22:28:55.247331 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:55.247259    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1691924-5354-4690-8709-6a20bfd81a7e-credential-socket\") pod \"e2e-inf-a-75fb62-openshift-default-c9cc6b459-4f2fg\" (UID: \"f1691924-5354-4690-8709-6a20bfd81a7e\") " pod="openshift-ingress/e2e-inf-a-75fb62-openshift-default-c9cc6b459-4f2fg"
Oct 30 22:28:55.247331 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:55.247327    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqwgx\" (UniqueName: \"kubernetes.io/projected/f1691924-5354-4690-8709-6a20bfd81a7e-kube-api-access-wqwgx\") pod \"e2e-inf-a-75fb62-openshift-default-c9cc6b459-4f2fg\" (UID: \"f1691924-5354-4690-8709-6a20bfd81a7e\") " pod="openshift-ingress/e2e-inf-a-75fb62-openshift-default-c9cc6b459-4f2fg"
Oct 30 22:28:55.247584 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:55.247435    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1691924-5354-4690-8709-6a20bfd81a7e-istiod-ca-cert\") pod \"e2e-inf-a-75fb62-openshift-default-c9cc6b459-4f2fg\" (UID: \"f1691924-5354-4690-8709-6a20bfd81a7e\") " pod="openshift-ingress/e2e-inf-a-75fb62-openshift-default-c9cc6b459-4f2fg"
Oct 30 22:28:55.247584 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:55.247486    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1691924-5354-4690-8709-6a20bfd81a7e-istio-token\") pod \"e2e-inf-a-75fb62-openshift-default-c9cc6b459-4f2fg\" (UID: \"f1691924-5354-4690-8709-6a20bfd81a7e\") " pod="openshift-ingress/e2e-inf-a-75fb62-openshift-default-c9cc6b459-4f2fg"
Oct 30 22:28:55.247584 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:55.247519    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1691924-5354-4690-8709-6a20bfd81a7e-istio-podinfo\") pod \"e2e-inf-a-75fb62-openshift-default-c9cc6b459-4f2fg\" (UID: \"f1691924-5354-4690-8709-6a20bfd81a7e\") " pod="openshift-ingress/e2e-inf-a-75fb62-openshift-default-c9cc6b459-4f2fg"
Oct 30 22:28:55.247584 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:55.247573    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1691924-5354-4690-8709-6a20bfd81a7e-workload-socket\") pod \"e2e-inf-a-75fb62-openshift-default-c9cc6b459-4f2fg\" (UID: \"f1691924-5354-4690-8709-6a20bfd81a7e\") " pod="openshift-ingress/e2e-inf-a-75fb62-openshift-default-c9cc6b459-4f2fg"
Oct 30 22:28:55.248046 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:55.248022    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1691924-5354-4690-8709-6a20bfd81a7e-workload-socket\") pod \"e2e-inf-a-75fb62-openshift-default-c9cc6b459-4f2fg\" (UID: \"f1691924-5354-4690-8709-6a20bfd81a7e\") " pod="openshift-ingress/e2e-inf-a-75fb62-openshift-default-c9cc6b459-4f2fg"
Oct 30 22:28:55.248759 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:55.248722    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1691924-5354-4690-8709-6a20bfd81a7e-istio-data\") pod \"e2e-inf-a-75fb62-openshift-default-c9cc6b459-4f2fg\" (UID: \"f1691924-5354-4690-8709-6a20bfd81a7e\") " pod="openshift-ingress/e2e-inf-a-75fb62-openshift-default-c9cc6b459-4f2fg"
Oct 30 22:28:55.248855 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:55.248838    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1691924-5354-4690-8709-6a20bfd81a7e-credential-socket\") pod \"e2e-inf-a-75fb62-openshift-default-c9cc6b459-4f2fg\" (UID: \"f1691924-5354-4690-8709-6a20bfd81a7e\") " pod="openshift-ingress/e2e-inf-a-75fb62-openshift-default-c9cc6b459-4f2fg"
Oct 30 22:28:55.248977 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:55.248953    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1691924-5354-4690-8709-6a20bfd81a7e-workload-certs\") pod \"e2e-inf-a-75fb62-openshift-default-c9cc6b459-4f2fg\" (UID: \"f1691924-5354-4690-8709-6a20bfd81a7e\") " pod="openshift-ingress/e2e-inf-a-75fb62-openshift-default-c9cc6b459-4f2fg"
Oct 30 22:28:55.249193 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:55.249167    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1691924-5354-4690-8709-6a20bfd81a7e-istiod-ca-cert\") pod \"e2e-inf-a-75fb62-openshift-default-c9cc6b459-4f2fg\" (UID: \"f1691924-5354-4690-8709-6a20bfd81a7e\") " pod="openshift-ingress/e2e-inf-a-75fb62-openshift-default-c9cc6b459-4f2fg"
Oct 30 22:28:55.251134 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:55.251114    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1691924-5354-4690-8709-6a20bfd81a7e-istio-envoy\") pod \"e2e-inf-a-75fb62-openshift-default-c9cc6b459-4f2fg\" (UID: \"f1691924-5354-4690-8709-6a20bfd81a7e\") " pod="openshift-ingress/e2e-inf-a-75fb62-openshift-default-c9cc6b459-4f2fg"
Oct 30 22:28:55.251221 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:55.251181    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1691924-5354-4690-8709-6a20bfd81a7e-istio-podinfo\") pod \"e2e-inf-a-75fb62-openshift-default-c9cc6b459-4f2fg\" (UID: \"f1691924-5354-4690-8709-6a20bfd81a7e\") " pod="openshift-ingress/e2e-inf-a-75fb62-openshift-default-c9cc6b459-4f2fg"
Oct 30 22:28:55.308964 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:55.308863    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1691924-5354-4690-8709-6a20bfd81a7e-istio-token\") pod \"e2e-inf-a-75fb62-openshift-default-c9cc6b459-4f2fg\" (UID: \"f1691924-5354-4690-8709-6a20bfd81a7e\") " pod="openshift-ingress/e2e-inf-a-75fb62-openshift-default-c9cc6b459-4f2fg"
Oct 30 22:28:55.309116 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:55.308995    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqwgx\" (UniqueName: \"kubernetes.io/projected/f1691924-5354-4690-8709-6a20bfd81a7e-kube-api-access-wqwgx\") pod \"e2e-inf-a-75fb62-openshift-default-c9cc6b459-4f2fg\" (UID: \"f1691924-5354-4690-8709-6a20bfd81a7e\") " pod="openshift-ingress/e2e-inf-a-75fb62-openshift-default-c9cc6b459-4f2fg"
Oct 30 22:28:55.357371 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:55.357340    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-75fb62-openshift-default-c9cc6b459-4f2fg"
Oct 30 22:28:55.810895 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:55.810860    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-75fb62-openshift-default-c9cc6b459-4f2fg"]
Oct 30 22:28:55.817212 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:28:55.817167    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1691924_5354_4690_8709_6a20bfd81a7e.slice/crio-53287d1705251e672c8b74eb6e3d3e046bed50ee982198749923cb37dd4e2b71 WatchSource:0}: Error finding container 53287d1705251e672c8b74eb6e3d3e046bed50ee982198749923cb37dd4e2b71: Status 404 returned error can't find the container with id 53287d1705251e672c8b74eb6e3d3e046bed50ee982198749923cb37dd4e2b71
Oct 30 22:28:55.820356 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:55.820321    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:28:55.820496 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:55.820425    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:28:55.820496 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:55.820472    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:28:56.762886 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:56.762851    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-75fb62-openshift-default-c9cc6b459-4f2fg" event={"ID":"f1691924-5354-4690-8709-6a20bfd81a7e","Type":"ContainerStarted","Data":"df99e50604825f2460a311054363d6eeaafae354297ddbb74bce51ae632392a6"}
Oct 30 22:28:56.763291 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:56.762910    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-75fb62-openshift-default-c9cc6b459-4f2fg" event={"ID":"f1691924-5354-4690-8709-6a20bfd81a7e","Type":"ContainerStarted","Data":"53287d1705251e672c8b74eb6e3d3e046bed50ee982198749923cb37dd4e2b71"}
Oct 30 22:28:56.928051 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:56.928000    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-75fb62-openshift-default-c9cc6b459-4f2fg" podStartSLOduration=2.927979485 podStartE2EDuration="2.927979485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:28:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:28:56.926183941 +0000 UTC m=+2314.530491493" watchObservedRunningTime="2026-10-30 22:28:56.927979485 +0000 UTC m=+2314.532287037"
Oct 30 22:28:57.358111 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:57.358077    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-75fb62-openshift-default-c9cc6b459-4f2fg"
Oct 30 22:28:57.364441 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:57.364416    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-75fb62-openshift-default-c9cc6b459-4f2fg"
Oct 30 22:28:57.769230 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:57.769203    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-75fb62-openshift-default-c9cc6b459-4f2fg"
Oct 30 22:28:57.770343 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:28:57.770323    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-75fb62-openshift-default-c9cc6b459-4f2fg"
Oct 30 22:29:00.131146 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:00.131106    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-65e59f-74d8d9b4c-t64kq" podUID="4b088ba1-b952-4a98-9316-7673c67ad3c8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:29:03.283778 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:03.283744    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-75fb62-57ff7485f9-fl2h5"]
Oct 30 22:29:03.284386 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:03.284369    5678 cpu_manager.go:414] "containerMap: removing container" podUID="9c5d0357-a8f0-484c-9584-4b0184fd5284" containerName="istio-proxy"
Oct 30 22:29:03.284386 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:03.284386    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c5d0357-a8f0-484c-9584-4b0184fd5284" containerName="istio-proxy"
Oct 30 22:29:03.288263 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:03.288242    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-75fb62-57ff7485f9-fl2h5"
Oct 30 22:29:03.355455 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:03.355386    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01b7d4f9-77ed-4292-a572-3a910bc8521e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-75fb62-57ff7485f9-fl2h5\" (UID: \"01b7d4f9-77ed-4292-a572-3a910bc8521e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-75fb62-57ff7485f9-fl2h5"
Oct 30 22:29:03.355602 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:03.355551    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26pd4\" (UniqueName: \"kubernetes.io/projected/01b7d4f9-77ed-4292-a572-3a910bc8521e-kube-api-access-26pd4\") pod \"payload-pre-processing-e2e-inf-a-75fb62-57ff7485f9-fl2h5\" (UID: \"01b7d4f9-77ed-4292-a572-3a910bc8521e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-75fb62-57ff7485f9-fl2h5"
Oct 30 22:29:03.385477 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:03.385452    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-75fb62-dockercfg-mspp5\""
Oct 30 22:29:03.385683 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:03.385469    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-75fb62\""
Oct 30 22:29:03.391142 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:03.391119    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-75fb62-57ff7485f9-fl2h5"]
Oct 30 22:29:03.488422 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:03.486469    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-75fb62-5649854489-92pfg"]
Oct 30 22:29:03.494434 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:03.492091    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26pd4\" (UniqueName: \"kubernetes.io/projected/01b7d4f9-77ed-4292-a572-3a910bc8521e-kube-api-access-26pd4\") pod \"payload-pre-processing-e2e-inf-a-75fb62-57ff7485f9-fl2h5\" (UID: \"01b7d4f9-77ed-4292-a572-3a910bc8521e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-75fb62-57ff7485f9-fl2h5"
Oct 30 22:29:03.494434 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:03.492377    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01b7d4f9-77ed-4292-a572-3a910bc8521e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-75fb62-57ff7485f9-fl2h5\" (UID: \"01b7d4f9-77ed-4292-a572-3a910bc8521e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-75fb62-57ff7485f9-fl2h5"
Oct 30 22:29:03.494434 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:03.493285    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01b7d4f9-77ed-4292-a572-3a910bc8521e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-75fb62-57ff7485f9-fl2h5\" (UID: \"01b7d4f9-77ed-4292-a572-3a910bc8521e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-75fb62-57ff7485f9-fl2h5"
Oct 30 22:29:03.571390 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:03.571298    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26pd4\" (UniqueName: \"kubernetes.io/projected/01b7d4f9-77ed-4292-a572-3a910bc8521e-kube-api-access-26pd4\") pod \"payload-pre-processing-e2e-inf-a-75fb62-57ff7485f9-fl2h5\" (UID: \"01b7d4f9-77ed-4292-a572-3a910bc8521e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-75fb62-57ff7485f9-fl2h5"
Oct 30 22:29:03.587147 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:03.587113    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-75fb62-5649854489-92pfg"]
Oct 30 22:29:03.587496 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:03.587474    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-75fb62-5649854489-92pfg"
Oct 30 22:29:03.603124 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:03.603087    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-75fb62-57ff7485f9-fl2h5"
Oct 30 22:29:03.694850 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:03.694814    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95ef4304-6677-4325-8396-31575317404c-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-75fb62-5649854489-92pfg\" (UID: \"95ef4304-6677-4325-8396-31575317404c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-75fb62-5649854489-92pfg"
Oct 30 22:29:03.695034 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:03.694943    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5g96\" (UniqueName: \"kubernetes.io/projected/95ef4304-6677-4325-8396-31575317404c-kube-api-access-d5g96\") pod \"payload-processing-e2e-inf-a-75fb62-5649854489-92pfg\" (UID: \"95ef4304-6677-4325-8396-31575317404c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-75fb62-5649854489-92pfg"
Oct 30 22:29:03.796579 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:03.796547    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95ef4304-6677-4325-8396-31575317404c-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-75fb62-5649854489-92pfg\" (UID: \"95ef4304-6677-4325-8396-31575317404c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-75fb62-5649854489-92pfg"
Oct 30 22:29:03.796755 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:03.796629    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5g96\" (UniqueName: \"kubernetes.io/projected/95ef4304-6677-4325-8396-31575317404c-kube-api-access-d5g96\") pod \"payload-processing-e2e-inf-a-75fb62-5649854489-92pfg\" (UID: \"95ef4304-6677-4325-8396-31575317404c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-75fb62-5649854489-92pfg"
Oct 30 22:29:03.797223 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:03.797201    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95ef4304-6677-4325-8396-31575317404c-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-75fb62-5649854489-92pfg\" (UID: \"95ef4304-6677-4325-8396-31575317404c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-75fb62-5649854489-92pfg"
Oct 30 22:29:03.855244 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:03.855178    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5g96\" (UniqueName: \"kubernetes.io/projected/95ef4304-6677-4325-8396-31575317404c-kube-api-access-d5g96\") pod \"payload-processing-e2e-inf-a-75fb62-5649854489-92pfg\" (UID: \"95ef4304-6677-4325-8396-31575317404c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-75fb62-5649854489-92pfg"
Oct 30 22:29:03.911213 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:03.911183    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-75fb62-5649854489-92pfg"
Oct 30 22:29:04.001277 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:04.001233    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-75fb62-57ff7485f9-fl2h5"]
Oct 30 22:29:04.024067 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:29:04.023707    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01b7d4f9_77ed_4292_a572_3a910bc8521e.slice/crio-8701985c82dcac1db69edfaeb7c183b66c73dc94baa5004f13854a03d6f88a0d WatchSource:0}: Error finding container 8701985c82dcac1db69edfaeb7c183b66c73dc94baa5004f13854a03d6f88a0d: Status 404 returned error can't find the container with id 8701985c82dcac1db69edfaeb7c183b66c73dc94baa5004f13854a03d6f88a0d
Oct 30 22:29:04.313452 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:04.313391    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-75fb62-5649854489-92pfg"]
Oct 30 22:29:04.319945 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:29:04.319918    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95ef4304_6677_4325_8396_31575317404c.slice/crio-b2c9bafd1a27d5cb74e5929a63a88b7f216a347be9df0d7c232aa8aa9d123af4 WatchSource:0}: Error finding container b2c9bafd1a27d5cb74e5929a63a88b7f216a347be9df0d7c232aa8aa9d123af4: Status 404 returned error can't find the container with id b2c9bafd1a27d5cb74e5929a63a88b7f216a347be9df0d7c232aa8aa9d123af4
Oct 30 22:29:04.818664 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:04.818626    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-75fb62-57ff7485f9-fl2h5" event={"ID":"01b7d4f9-77ed-4292-a572-3a910bc8521e","Type":"ContainerStarted","Data":"708b64baddc2018a602c41d286ea7bb816b507bc0119e5f035dd3bafa3d7336a"}
Oct 30 22:29:04.818897 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:04.818691    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-75fb62-57ff7485f9-fl2h5" event={"ID":"01b7d4f9-77ed-4292-a572-3a910bc8521e","Type":"ContainerStarted","Data":"8701985c82dcac1db69edfaeb7c183b66c73dc94baa5004f13854a03d6f88a0d"}
Oct 30 22:29:04.818897 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:04.818825    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-75fb62-57ff7485f9-fl2h5"
Oct 30 22:29:04.820452 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:04.820428    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-75fb62-5649854489-92pfg" event={"ID":"95ef4304-6677-4325-8396-31575317404c","Type":"ContainerStarted","Data":"e04707050b9f01b8187bff71c74d10dcfd51ffa90ff37773891da0a9e02b6c7e"}
Oct 30 22:29:04.820573 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:04.820478    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-75fb62-5649854489-92pfg"
Oct 30 22:29:04.820573 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:04.820502    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-75fb62-5649854489-92pfg" event={"ID":"95ef4304-6677-4325-8396-31575317404c","Type":"ContainerStarted","Data":"b2c9bafd1a27d5cb74e5929a63a88b7f216a347be9df0d7c232aa8aa9d123af4"}
Oct 30 22:29:04.930463 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:04.930372    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-75fb62-57ff7485f9-fl2h5" podStartSLOduration=1.930354338 podStartE2EDuration="1.930354338s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:29:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:29:04.930097667 +0000 UTC m=+2322.534405220" watchObservedRunningTime="2026-10-30 22:29:04.930354338 +0000 UTC m=+2322.534661891"
Oct 30 22:29:05.047070 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:05.046973    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-75fb62-5649854489-92pfg" podStartSLOduration=2.046952855 podStartE2EDuration="2.046952855s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:29:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:29:05.04511042 +0000 UTC m=+2322.649417984" watchObservedRunningTime="2026-10-30 22:29:05.046952855 +0000 UTC m=+2322.651260410"
Oct 30 22:29:09.464132 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:09.464072    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-65e59f-74d8d9b4c-t64kq"
Oct 30 22:29:09.512418 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:09.512374    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b088ba1-b952-4a98-9316-7673c67ad3c8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b088ba1-b952-4a98-9316-7673c67ad3c8-plugins-config-volume\") pod \"4b088ba1-b952-4a98-9316-7673c67ad3c8\" (UID: \"4b088ba1-b952-4a98-9316-7673c67ad3c8\") "
Oct 30 22:29:09.512584 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:09.512466    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b088ba1-b952-4a98-9316-7673c67ad3c8-kube-api-access-qlrxg\" (UniqueName: \"kubernetes.io/projected/4b088ba1-b952-4a98-9316-7673c67ad3c8-kube-api-access-qlrxg\") pod \"4b088ba1-b952-4a98-9316-7673c67ad3c8\" (UID: \"4b088ba1-b952-4a98-9316-7673c67ad3c8\") "
Oct 30 22:29:09.512877 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:09.512837    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b088ba1-b952-4a98-9316-7673c67ad3c8-plugins-config-volume" pod "4b088ba1-b952-4a98-9316-7673c67ad3c8" (UID: "4b088ba1-b952-4a98-9316-7673c67ad3c8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:29:09.513180 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:09.513156    5678 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b088ba1-b952-4a98-9316-7673c67ad3c8-plugins-config-volume\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:09.515183 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:09.515158    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b088ba1-b952-4a98-9316-7673c67ad3c8-kube-api-access-qlrxg" pod "4b088ba1-b952-4a98-9316-7673c67ad3c8" (UID: "4b088ba1-b952-4a98-9316-7673c67ad3c8"). InnerVolumeSpecName "kube-api-access-qlrxg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:29:09.614878 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:09.614845    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlrxg\" (UniqueName: \"kubernetes.io/projected/4b088ba1-b952-4a98-9316-7673c67ad3c8-kube-api-access-qlrxg\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:09.857115 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:09.857027    5678 generic.go:356] "Generic (PLEG): container finished" podID="4b088ba1-b952-4a98-9316-7673c67ad3c8" containerID="8d7f8b4b344dbd3851a421b5cabafb83cdc35454617ae8b7c82b8e720979bcda" exitCode=0
Oct 30 22:29:09.857115 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:09.857061    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-65e59f-74d8d9b4c-t64kq" event={"ID":"4b088ba1-b952-4a98-9316-7673c67ad3c8","Type":"ContainerDied","Data":"8d7f8b4b344dbd3851a421b5cabafb83cdc35454617ae8b7c82b8e720979bcda"}
Oct 30 22:29:09.857115 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:09.857080    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-65e59f-74d8d9b4c-t64kq"
Oct 30 22:29:09.857390 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:09.857121    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-65e59f-74d8d9b4c-t64kq" event={"ID":"4b088ba1-b952-4a98-9316-7673c67ad3c8","Type":"ContainerDied","Data":"7655793ef295df75503fd9705e83b77986f31ba5d23b7b09f81bac92089119db"}
Oct 30 22:29:09.857390 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:09.857145    5678 scope.go:122] "RemoveContainer" containerID="8d7f8b4b344dbd3851a421b5cabafb83cdc35454617ae8b7c82b8e720979bcda"
Oct 30 22:29:09.867748 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:09.867731    5678 scope.go:122] "RemoveContainer" containerID="8d7f8b4b344dbd3851a421b5cabafb83cdc35454617ae8b7c82b8e720979bcda"
Oct 30 22:29:09.868000 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:29:09.867976    5678 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d7f8b4b344dbd3851a421b5cabafb83cdc35454617ae8b7c82b8e720979bcda\": container with ID starting with 8d7f8b4b344dbd3851a421b5cabafb83cdc35454617ae8b7c82b8e720979bcda not found: ID does not exist" containerID="8d7f8b4b344dbd3851a421b5cabafb83cdc35454617ae8b7c82b8e720979bcda"
Oct 30 22:29:09.868051 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:09.868011    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d7f8b4b344dbd3851a421b5cabafb83cdc35454617ae8b7c82b8e720979bcda"} err="failed to get container status \"8d7f8b4b344dbd3851a421b5cabafb83cdc35454617ae8b7c82b8e720979bcda\": rpc error: code = NotFound desc = could not find container \"8d7f8b4b344dbd3851a421b5cabafb83cdc35454617ae8b7c82b8e720979bcda\": container with ID starting with 8d7f8b4b344dbd3851a421b5cabafb83cdc35454617ae8b7c82b8e720979bcda not found: ID does not exist"
Oct 30 22:29:10.014983 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:10.014945    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-65e59f-74d8d9b4c-t64kq"]
Oct 30 22:29:10.019391 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:10.019361    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-65e59f-74d8d9b4c-t64kq"]
Oct 30 22:29:10.370666 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:10.370631    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-595fd7d7-656b887dc9-5fgv6"]
Oct 30 22:29:10.370897 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:10.370855    5678 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-595fd7d7-656b887dc9-5fgv6" podUID="7c5d99a1-0c2f-4e44-87d0-3da5c6ee4a6d" containerName="maas-api" containerID="cri-o://b23b131e739340e130c7c41ba801b565b831ae7101cc5e65cb1ea8791607b578" gracePeriod=30
Oct 30 22:29:10.526213 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:10.526178    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b088ba1-b952-4a98-9316-7673c67ad3c8" path="/var/lib/kubelet/pods/4b088ba1-b952-4a98-9316-7673c67ad3c8/volumes"
Oct 30 22:29:10.595102 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:29:10.595063    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 22:29:10.676173 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:10.676142    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-595fd7d7-656b887dc9-5fgv6"
Oct 30 22:29:10.739533 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:10.739506    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c5d99a1-0c2f-4e44-87d0-3da5c6ee4a6d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c5d99a1-0c2f-4e44-87d0-3da5c6ee4a6d-openshift-service-ca\") pod \"7c5d99a1-0c2f-4e44-87d0-3da5c6ee4a6d\" (UID: \"7c5d99a1-0c2f-4e44-87d0-3da5c6ee4a6d\") "
Oct 30 22:29:10.739680 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:10.739635    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c5d99a1-0c2f-4e44-87d0-3da5c6ee4a6d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c5d99a1-0c2f-4e44-87d0-3da5c6ee4a6d-metrics-certs\") pod \"7c5d99a1-0c2f-4e44-87d0-3da5c6ee4a6d\" (UID: \"7c5d99a1-0c2f-4e44-87d0-3da5c6ee4a6d\") "
Oct 30 22:29:10.739753 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:10.739735    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c5d99a1-0c2f-4e44-87d0-3da5c6ee4a6d-kube-api-access-7khcg\" (UniqueName: \"kubernetes.io/projected/7c5d99a1-0c2f-4e44-87d0-3da5c6ee4a6d-kube-api-access-7khcg\") pod \"7c5d99a1-0c2f-4e44-87d0-3da5c6ee4a6d\" (UID: \"7c5d99a1-0c2f-4e44-87d0-3da5c6ee4a6d\") "
Oct 30 22:29:10.739827 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:10.739809    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c5d99a1-0c2f-4e44-87d0-3da5c6ee4a6d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c5d99a1-0c2f-4e44-87d0-3da5c6ee4a6d-maas-api-tls\") pod \"7c5d99a1-0c2f-4e44-87d0-3da5c6ee4a6d\" (UID: \"7c5d99a1-0c2f-4e44-87d0-3da5c6ee4a6d\") "
Oct 30 22:29:10.739914 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:10.739892    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c5d99a1-0c2f-4e44-87d0-3da5c6ee4a6d-openshift-service-ca" pod "7c5d99a1-0c2f-4e44-87d0-3da5c6ee4a6d" (UID: "7c5d99a1-0c2f-4e44-87d0-3da5c6ee4a6d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:29:10.740433 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:10.740385    5678 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c5d99a1-0c2f-4e44-87d0-3da5c6ee4a6d-openshift-service-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:10.741833 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:10.741778    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c5d99a1-0c2f-4e44-87d0-3da5c6ee4a6d-metrics-certs" pod "7c5d99a1-0c2f-4e44-87d0-3da5c6ee4a6d" (UID: "7c5d99a1-0c2f-4e44-87d0-3da5c6ee4a6d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:29:10.741990 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:10.741959    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c5d99a1-0c2f-4e44-87d0-3da5c6ee4a6d-kube-api-access-7khcg" pod "7c5d99a1-0c2f-4e44-87d0-3da5c6ee4a6d" (UID: "7c5d99a1-0c2f-4e44-87d0-3da5c6ee4a6d"). InnerVolumeSpecName "kube-api-access-7khcg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:29:10.742314 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:10.742287    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c5d99a1-0c2f-4e44-87d0-3da5c6ee4a6d-maas-api-tls" pod "7c5d99a1-0c2f-4e44-87d0-3da5c6ee4a6d" (UID: "7c5d99a1-0c2f-4e44-87d0-3da5c6ee4a6d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:29:10.841898 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:10.841860    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c5d99a1-0c2f-4e44-87d0-3da5c6ee4a6d-maas-api-tls\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:10.841898 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:10.841896    5678 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c5d99a1-0c2f-4e44-87d0-3da5c6ee4a6d-metrics-certs\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:10.842119 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:10.841907    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7khcg\" (UniqueName: \"kubernetes.io/projected/7c5d99a1-0c2f-4e44-87d0-3da5c6ee4a6d-kube-api-access-7khcg\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:10.874421 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:10.872249    5678 generic.go:356] "Generic (PLEG): container finished" podID="7c5d99a1-0c2f-4e44-87d0-3da5c6ee4a6d" containerID="b23b131e739340e130c7c41ba801b565b831ae7101cc5e65cb1ea8791607b578" exitCode=0
Oct 30 22:29:10.874421 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:10.872324    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-595fd7d7-656b887dc9-5fgv6" event={"ID":"7c5d99a1-0c2f-4e44-87d0-3da5c6ee4a6d","Type":"ContainerDied","Data":"b23b131e739340e130c7c41ba801b565b831ae7101cc5e65cb1ea8791607b578"}
Oct 30 22:29:10.874421 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:10.872387    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-595fd7d7-656b887dc9-5fgv6" event={"ID":"7c5d99a1-0c2f-4e44-87d0-3da5c6ee4a6d","Type":"ContainerDied","Data":"c2df2c0f43b6682cadf073362da8ca44972f40c3f18b95e80522e15b817cfc80"}
Oct 30 22:29:10.874421 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:10.872435    5678 scope.go:122] "RemoveContainer" containerID="b23b131e739340e130c7c41ba801b565b831ae7101cc5e65cb1ea8791607b578"
Oct 30 22:29:10.874421 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:10.872466    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-595fd7d7-656b887dc9-5fgv6"
Oct 30 22:29:10.898536 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:10.898512    5678 scope.go:122] "RemoveContainer" containerID="b23b131e739340e130c7c41ba801b565b831ae7101cc5e65cb1ea8791607b578"
Oct 30 22:29:10.898958 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:29:10.898895    5678 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b23b131e739340e130c7c41ba801b565b831ae7101cc5e65cb1ea8791607b578\": container with ID starting with b23b131e739340e130c7c41ba801b565b831ae7101cc5e65cb1ea8791607b578 not found: ID does not exist" containerID="b23b131e739340e130c7c41ba801b565b831ae7101cc5e65cb1ea8791607b578"
Oct 30 22:29:10.898958 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:10.898928    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b23b131e739340e130c7c41ba801b565b831ae7101cc5e65cb1ea8791607b578"} err="failed to get container status \"b23b131e739340e130c7c41ba801b565b831ae7101cc5e65cb1ea8791607b578\": rpc error: code = NotFound desc = could not find container \"b23b131e739340e130c7c41ba801b565b831ae7101cc5e65cb1ea8791607b578\": container with ID starting with b23b131e739340e130c7c41ba801b565b831ae7101cc5e65cb1ea8791607b578 not found: ID does not exist"
Oct 30 22:29:11.036087 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:11.036053    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-595fd7d7-656b887dc9-5fgv6"]
Oct 30 22:29:11.039232 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:11.039210    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-595fd7d7-656b887dc9-5fgv6"]
Oct 30 22:29:12.526551 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:12.526515    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c5d99a1-0c2f-4e44-87d0-3da5c6ee4a6d" path="/var/lib/kubelet/pods/7c5d99a1-0c2f-4e44-87d0-3da5c6ee4a6d/volumes"
Oct 30 22:29:15.837297 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:15.837261    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-75fb62-57ff7485f9-fl2h5"
Oct 30 22:29:15.837990 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:15.837365    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-75fb62-5649854489-92pfg"
Oct 30 22:29:23.520957 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:23.520928    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-582593-openshift-default-868f8d8b7-w97qr"]
Oct 30 22:29:23.521566 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:23.521544    5678 cpu_manager.go:414] "containerMap: removing container" podUID="4b088ba1-b952-4a98-9316-7673c67ad3c8" containerName="payload-pre-processing"
Oct 30 22:29:23.521566 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:23.521561    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b088ba1-b952-4a98-9316-7673c67ad3c8" containerName="payload-pre-processing"
Oct 30 22:29:23.521566 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:23.521572    5678 cpu_manager.go:414] "containerMap: removing container" podUID="7c5d99a1-0c2f-4e44-87d0-3da5c6ee4a6d" containerName="maas-api"
Oct 30 22:29:23.521781 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:23.521578    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c5d99a1-0c2f-4e44-87d0-3da5c6ee4a6d" containerName="maas-api"
Oct 30 22:29:23.521781 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:23.521710    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b088ba1-b952-4a98-9316-7673c67ad3c8" containerName="payload-pre-processing"
Oct 30 22:29:23.521781 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:23.521720    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c5d99a1-0c2f-4e44-87d0-3da5c6ee4a6d" containerName="maas-api"
Oct 30 22:29:23.525705 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:23.525687    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-582593-openshift-default-868f8d8b7-w97qr"
Oct 30 22:29:23.550873 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:23.550839    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsfrx\" (UniqueName: \"kubernetes.io/projected/e325caa7-d4fb-4a0e-9fe0-ee95742278b3-kube-api-access-rsfrx\") pod \"e2e-inf-a-582593-openshift-default-868f8d8b7-w97qr\" (UID: \"e325caa7-d4fb-4a0e-9fe0-ee95742278b3\") " pod="openshift-ingress/e2e-inf-a-582593-openshift-default-868f8d8b7-w97qr"
Oct 30 22:29:23.551176 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:23.551154    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e325caa7-d4fb-4a0e-9fe0-ee95742278b3-istio-data\") pod \"e2e-inf-a-582593-openshift-default-868f8d8b7-w97qr\" (UID: \"e325caa7-d4fb-4a0e-9fe0-ee95742278b3\") " pod="openshift-ingress/e2e-inf-a-582593-openshift-default-868f8d8b7-w97qr"
Oct 30 22:29:23.566584 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:23.566551    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e325caa7-d4fb-4a0e-9fe0-ee95742278b3-istiod-ca-cert\") pod \"e2e-inf-a-582593-openshift-default-868f8d8b7-w97qr\" (UID: \"e325caa7-d4fb-4a0e-9fe0-ee95742278b3\") " pod="openshift-ingress/e2e-inf-a-582593-openshift-default-868f8d8b7-w97qr"
Oct 30 22:29:23.566837 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:23.566819    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e325caa7-d4fb-4a0e-9fe0-ee95742278b3-workload-certs\") pod \"e2e-inf-a-582593-openshift-default-868f8d8b7-w97qr\" (UID: \"e325caa7-d4fb-4a0e-9fe0-ee95742278b3\") " pod="openshift-ingress/e2e-inf-a-582593-openshift-default-868f8d8b7-w97qr"
Oct 30 22:29:23.566971 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:23.566957    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e325caa7-d4fb-4a0e-9fe0-ee95742278b3-credential-socket\") pod \"e2e-inf-a-582593-openshift-default-868f8d8b7-w97qr\" (UID: \"e325caa7-d4fb-4a0e-9fe0-ee95742278b3\") " pod="openshift-ingress/e2e-inf-a-582593-openshift-default-868f8d8b7-w97qr"
Oct 30 22:29:23.569430 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:23.567072    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e325caa7-d4fb-4a0e-9fe0-ee95742278b3-istio-envoy\") pod \"e2e-inf-a-582593-openshift-default-868f8d8b7-w97qr\" (UID: \"e325caa7-d4fb-4a0e-9fe0-ee95742278b3\") " pod="openshift-ingress/e2e-inf-a-582593-openshift-default-868f8d8b7-w97qr"
Oct 30 22:29:23.580421 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:23.569848    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e325caa7-d4fb-4a0e-9fe0-ee95742278b3-workload-socket\") pod \"e2e-inf-a-582593-openshift-default-868f8d8b7-w97qr\" (UID: \"e325caa7-d4fb-4a0e-9fe0-ee95742278b3\") " pod="openshift-ingress/e2e-inf-a-582593-openshift-default-868f8d8b7-w97qr"
Oct 30 22:29:23.580656 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:23.580631    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e325caa7-d4fb-4a0e-9fe0-ee95742278b3-istio-token\") pod \"e2e-inf-a-582593-openshift-default-868f8d8b7-w97qr\" (UID: \"e325caa7-d4fb-4a0e-9fe0-ee95742278b3\") " pod="openshift-ingress/e2e-inf-a-582593-openshift-default-868f8d8b7-w97qr"
Oct 30 22:29:23.599420 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:23.580803    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e325caa7-d4fb-4a0e-9fe0-ee95742278b3-istio-podinfo\") pod \"e2e-inf-a-582593-openshift-default-868f8d8b7-w97qr\" (UID: \"e325caa7-d4fb-4a0e-9fe0-ee95742278b3\") " pod="openshift-ingress/e2e-inf-a-582593-openshift-default-868f8d8b7-w97qr"
Oct 30 22:29:23.613890 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:23.588844    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-582593-openshift-default-dockercfg-vwzjz\""
Oct 30 22:29:23.666928 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:23.666874    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-582593-openshift-default-868f8d8b7-w97qr"]
Oct 30 22:29:23.717096 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:23.717057    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e325caa7-d4fb-4a0e-9fe0-ee95742278b3-istio-podinfo\") pod \"e2e-inf-a-582593-openshift-default-868f8d8b7-w97qr\" (UID: \"e325caa7-d4fb-4a0e-9fe0-ee95742278b3\") " pod="openshift-ingress/e2e-inf-a-582593-openshift-default-868f8d8b7-w97qr"
Oct 30 22:29:23.717453 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:23.717427    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsfrx\" (UniqueName: \"kubernetes.io/projected/e325caa7-d4fb-4a0e-9fe0-ee95742278b3-kube-api-access-rsfrx\") pod \"e2e-inf-a-582593-openshift-default-868f8d8b7-w97qr\" (UID: \"e325caa7-d4fb-4a0e-9fe0-ee95742278b3\") " pod="openshift-ingress/e2e-inf-a-582593-openshift-default-868f8d8b7-w97qr"
Oct 30 22:29:23.717660 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:23.717642    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e325caa7-d4fb-4a0e-9fe0-ee95742278b3-istio-data\") pod \"e2e-inf-a-582593-openshift-default-868f8d8b7-w97qr\" (UID: \"e325caa7-d4fb-4a0e-9fe0-ee95742278b3\") " pod="openshift-ingress/e2e-inf-a-582593-openshift-default-868f8d8b7-w97qr"
Oct 30 22:29:23.717788 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:23.717772    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e325caa7-d4fb-4a0e-9fe0-ee95742278b3-istiod-ca-cert\") pod \"e2e-inf-a-582593-openshift-default-868f8d8b7-w97qr\" (UID: \"e325caa7-d4fb-4a0e-9fe0-ee95742278b3\") " pod="openshift-ingress/e2e-inf-a-582593-openshift-default-868f8d8b7-w97qr"
Oct 30 22:29:23.717925 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:23.717911    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e325caa7-d4fb-4a0e-9fe0-ee95742278b3-workload-certs\") pod \"e2e-inf-a-582593-openshift-default-868f8d8b7-w97qr\" (UID: \"e325caa7-d4fb-4a0e-9fe0-ee95742278b3\") " pod="openshift-ingress/e2e-inf-a-582593-openshift-default-868f8d8b7-w97qr"
Oct 30 22:29:23.718035 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:23.718021    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e325caa7-d4fb-4a0e-9fe0-ee95742278b3-credential-socket\") pod \"e2e-inf-a-582593-openshift-default-868f8d8b7-w97qr\" (UID: \"e325caa7-d4fb-4a0e-9fe0-ee95742278b3\") " pod="openshift-ingress/e2e-inf-a-582593-openshift-default-868f8d8b7-w97qr"
Oct 30 22:29:23.718159 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:23.718144    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e325caa7-d4fb-4a0e-9fe0-ee95742278b3-istio-envoy\") pod \"e2e-inf-a-582593-openshift-default-868f8d8b7-w97qr\" (UID: \"e325caa7-d4fb-4a0e-9fe0-ee95742278b3\") " pod="openshift-ingress/e2e-inf-a-582593-openshift-default-868f8d8b7-w97qr"
Oct 30 22:29:23.718455 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:23.718434    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e325caa7-d4fb-4a0e-9fe0-ee95742278b3-workload-socket\") pod \"e2e-inf-a-582593-openshift-default-868f8d8b7-w97qr\" (UID: \"e325caa7-d4fb-4a0e-9fe0-ee95742278b3\") " pod="openshift-ingress/e2e-inf-a-582593-openshift-default-868f8d8b7-w97qr"
Oct 30 22:29:23.718600 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:23.718582    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e325caa7-d4fb-4a0e-9fe0-ee95742278b3-istio-token\") pod \"e2e-inf-a-582593-openshift-default-868f8d8b7-w97qr\" (UID: \"e325caa7-d4fb-4a0e-9fe0-ee95742278b3\") " pod="openshift-ingress/e2e-inf-a-582593-openshift-default-868f8d8b7-w97qr"
Oct 30 22:29:23.720358 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:23.720209    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e325caa7-d4fb-4a0e-9fe0-ee95742278b3-istio-data\") pod \"e2e-inf-a-582593-openshift-default-868f8d8b7-w97qr\" (UID: \"e325caa7-d4fb-4a0e-9fe0-ee95742278b3\") " pod="openshift-ingress/e2e-inf-a-582593-openshift-default-868f8d8b7-w97qr"
Oct 30 22:29:23.721758 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:23.721533    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e325caa7-d4fb-4a0e-9fe0-ee95742278b3-credential-socket\") pod \"e2e-inf-a-582593-openshift-default-868f8d8b7-w97qr\" (UID: \"e325caa7-d4fb-4a0e-9fe0-ee95742278b3\") " pod="openshift-ingress/e2e-inf-a-582593-openshift-default-868f8d8b7-w97qr"
Oct 30 22:29:23.722652 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:23.722585    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e325caa7-d4fb-4a0e-9fe0-ee95742278b3-istiod-ca-cert\") pod \"e2e-inf-a-582593-openshift-default-868f8d8b7-w97qr\" (UID: \"e325caa7-d4fb-4a0e-9fe0-ee95742278b3\") " pod="openshift-ingress/e2e-inf-a-582593-openshift-default-868f8d8b7-w97qr"
Oct 30 22:29:23.723008 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:23.722968    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e325caa7-d4fb-4a0e-9fe0-ee95742278b3-workload-socket\") pod \"e2e-inf-a-582593-openshift-default-868f8d8b7-w97qr\" (UID: \"e325caa7-d4fb-4a0e-9fe0-ee95742278b3\") " pod="openshift-ingress/e2e-inf-a-582593-openshift-default-868f8d8b7-w97qr"
Oct 30 22:29:23.723192 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:23.723156    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e325caa7-d4fb-4a0e-9fe0-ee95742278b3-workload-certs\") pod \"e2e-inf-a-582593-openshift-default-868f8d8b7-w97qr\" (UID: \"e325caa7-d4fb-4a0e-9fe0-ee95742278b3\") " pod="openshift-ingress/e2e-inf-a-582593-openshift-default-868f8d8b7-w97qr"
Oct 30 22:29:23.750933 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:23.750877    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e325caa7-d4fb-4a0e-9fe0-ee95742278b3-istio-envoy\") pod \"e2e-inf-a-582593-openshift-default-868f8d8b7-w97qr\" (UID: \"e325caa7-d4fb-4a0e-9fe0-ee95742278b3\") " pod="openshift-ingress/e2e-inf-a-582593-openshift-default-868f8d8b7-w97qr"
Oct 30 22:29:23.760506 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:23.760470    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e325caa7-d4fb-4a0e-9fe0-ee95742278b3-istio-podinfo\") pod \"e2e-inf-a-582593-openshift-default-868f8d8b7-w97qr\" (UID: \"e325caa7-d4fb-4a0e-9fe0-ee95742278b3\") " pod="openshift-ingress/e2e-inf-a-582593-openshift-default-868f8d8b7-w97qr"
Oct 30 22:29:23.778395 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:23.778323    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e325caa7-d4fb-4a0e-9fe0-ee95742278b3-istio-token\") pod \"e2e-inf-a-582593-openshift-default-868f8d8b7-w97qr\" (UID: \"e325caa7-d4fb-4a0e-9fe0-ee95742278b3\") " pod="openshift-ingress/e2e-inf-a-582593-openshift-default-868f8d8b7-w97qr"
Oct 30 22:29:23.778395 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:23.778384    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsfrx\" (UniqueName: \"kubernetes.io/projected/e325caa7-d4fb-4a0e-9fe0-ee95742278b3-kube-api-access-rsfrx\") pod \"e2e-inf-a-582593-openshift-default-868f8d8b7-w97qr\" (UID: \"e325caa7-d4fb-4a0e-9fe0-ee95742278b3\") " pod="openshift-ingress/e2e-inf-a-582593-openshift-default-868f8d8b7-w97qr"
Oct 30 22:29:23.840109 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:23.840081    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-582593-openshift-default-868f8d8b7-w97qr"
Oct 30 22:29:24.298783 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:24.298751    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-582593-openshift-default-868f8d8b7-w97qr"]
Oct 30 22:29:24.345774 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:29:24.345742    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode325caa7_d4fb_4a0e_9fe0_ee95742278b3.slice/crio-a1e254a1f816b22abb5964e9fdf8c0e1a38f2b6f3cd6ab9295d1adbfac957881 WatchSource:0}: Error finding container a1e254a1f816b22abb5964e9fdf8c0e1a38f2b6f3cd6ab9295d1adbfac957881: Status 404 returned error can't find the container with id a1e254a1f816b22abb5964e9fdf8c0e1a38f2b6f3cd6ab9295d1adbfac957881
Oct 30 22:29:24.348370 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:24.348340    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:29:24.348462 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:24.348443    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:29:24.348506 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:24.348479    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:29:24.988097 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:24.988066    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-582593-openshift-default-868f8d8b7-w97qr" event={"ID":"e325caa7-d4fb-4a0e-9fe0-ee95742278b3","Type":"ContainerStarted","Data":"bd50cf584c9705bd333a3688197936c0b11b4611c0c209d3b004442718f52b1e"}
Oct 30 22:29:24.988569 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:24.988114    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-582593-openshift-default-868f8d8b7-w97qr" event={"ID":"e325caa7-d4fb-4a0e-9fe0-ee95742278b3","Type":"ContainerStarted","Data":"a1e254a1f816b22abb5964e9fdf8c0e1a38f2b6f3cd6ab9295d1adbfac957881"}
Oct 30 22:29:25.154023 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:25.153967    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-582593-openshift-default-868f8d8b7-w97qr" podStartSLOduration=2.153952575 podStartE2EDuration="2.153952575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:29:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:29:25.151210082 +0000 UTC m=+2342.755517635" watchObservedRunningTime="2026-10-30 22:29:25.153952575 +0000 UTC m=+2342.758260128"
Oct 30 22:29:25.841249 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:25.841221    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-582593-openshift-default-868f8d8b7-w97qr"
Oct 30 22:29:25.846768 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:25.846741    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-582593-openshift-default-868f8d8b7-w97qr"
Oct 30 22:29:25.994756 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:25.994726    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-582593-openshift-default-868f8d8b7-w97qr"
Oct 30 22:29:25.995839 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:25.995814    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-582593-openshift-default-868f8d8b7-w97qr"
Oct 30 22:29:31.662452 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:31.662352    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-582593-748c54b589-7kjl4"]
Oct 30 22:29:31.667107 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:31.667082    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-582593-748c54b589-7kjl4"
Oct 30 22:29:31.721829 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:31.721778    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-582593-748c54b589-7kjl4"]
Oct 30 22:29:31.764600 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:31.764568    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-582593-dockercfg-t9bjg\""
Oct 30 22:29:31.764600 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:31.764568    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-582593\""
Oct 30 22:29:31.834275 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:31.834243    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvxvb\" (UniqueName: \"kubernetes.io/projected/b1db85b2-f8d2-42b9-8731-4e9e9bff3f38-kube-api-access-rvxvb\") pod \"payload-processing-e2e-inf-a-582593-748c54b589-7kjl4\" (UID: \"b1db85b2-f8d2-42b9-8731-4e9e9bff3f38\") " pod="openshift-ingress/payload-processing-e2e-inf-a-582593-748c54b589-7kjl4"
Oct 30 22:29:31.834495 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:31.834295    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1db85b2-f8d2-42b9-8731-4e9e9bff3f38-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-582593-748c54b589-7kjl4\" (UID: \"b1db85b2-f8d2-42b9-8731-4e9e9bff3f38\") " pod="openshift-ingress/payload-processing-e2e-inf-a-582593-748c54b589-7kjl4"
Oct 30 22:29:31.949472 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:31.946365    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvxvb\" (UniqueName: \"kubernetes.io/projected/b1db85b2-f8d2-42b9-8731-4e9e9bff3f38-kube-api-access-rvxvb\") pod \"payload-processing-e2e-inf-a-582593-748c54b589-7kjl4\" (UID: \"b1db85b2-f8d2-42b9-8731-4e9e9bff3f38\") " pod="openshift-ingress/payload-processing-e2e-inf-a-582593-748c54b589-7kjl4"
Oct 30 22:29:31.949472 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:31.946893    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1db85b2-f8d2-42b9-8731-4e9e9bff3f38-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-582593-748c54b589-7kjl4\" (UID: \"b1db85b2-f8d2-42b9-8731-4e9e9bff3f38\") " pod="openshift-ingress/payload-processing-e2e-inf-a-582593-748c54b589-7kjl4"
Oct 30 22:29:31.949472 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:31.947941    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1db85b2-f8d2-42b9-8731-4e9e9bff3f38-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-582593-748c54b589-7kjl4\" (UID: \"b1db85b2-f8d2-42b9-8731-4e9e9bff3f38\") " pod="openshift-ingress/payload-processing-e2e-inf-a-582593-748c54b589-7kjl4"
Oct 30 22:29:32.006854 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:32.006821    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvxvb\" (UniqueName: \"kubernetes.io/projected/b1db85b2-f8d2-42b9-8731-4e9e9bff3f38-kube-api-access-rvxvb\") pod \"payload-processing-e2e-inf-a-582593-748c54b589-7kjl4\" (UID: \"b1db85b2-f8d2-42b9-8731-4e9e9bff3f38\") " pod="openshift-ingress/payload-processing-e2e-inf-a-582593-748c54b589-7kjl4"
Oct 30 22:29:32.284760 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:32.284662    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-582593-748c54b589-7kjl4"
Oct 30 22:29:32.768970 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:32.768931    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-582593-748c54b589-7kjl4"]
Oct 30 22:29:32.775513 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:29:32.775483    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1db85b2_f8d2_42b9_8731_4e9e9bff3f38.slice/crio-f004f14d5155b55de750313d6206d9199b8a0b842231db206ff5fc521c62b9e5 WatchSource:0}: Error finding container f004f14d5155b55de750313d6206d9199b8a0b842231db206ff5fc521c62b9e5: Status 404 returned error can't find the container with id f004f14d5155b55de750313d6206d9199b8a0b842231db206ff5fc521c62b9e5
Oct 30 22:29:33.047658 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:33.047568    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-582593-748c54b589-7kjl4" event={"ID":"b1db85b2-f8d2-42b9-8731-4e9e9bff3f38","Type":"ContainerStarted","Data":"4a802be0a5fa7992a0744ffa341d40593c2f85b641575dee1ff897a3e214cb25"}
Oct 30 22:29:33.047658 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:33.047630    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-582593-748c54b589-7kjl4" event={"ID":"b1db85b2-f8d2-42b9-8731-4e9e9bff3f38","Type":"ContainerStarted","Data":"f004f14d5155b55de750313d6206d9199b8a0b842231db206ff5fc521c62b9e5"}
Oct 30 22:29:33.047658 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:33.047657    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-582593-748c54b589-7kjl4"
Oct 30 22:29:33.162512 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:33.162456    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-582593-748c54b589-7kjl4" podStartSLOduration=2.16244199 podStartE2EDuration="2.16244199s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:29:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:29:33.160269931 +0000 UTC m=+2350.764577485" watchObservedRunningTime="2026-10-30 22:29:33.16244199 +0000 UTC m=+2350.766749543"
Oct 30 22:29:40.579898 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:29:40.579857    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 22:29:40.589270 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:40.589243    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-3cc1b3-openshift-default-744b6b9d7f-lx8n5"]
Oct 30 22:29:40.589532 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:40.589507    5678 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w5-3cc1b3-openshift-default-744b6b9d7f-lx8n5" podUID="4b6efa36-7840-418b-be4a-723ca4407a91" containerName="istio-proxy" containerID="cri-o://d21d441a96db7f14edbbc29043f50740221659824026888da3bcaf666bc476de" gracePeriod=30
Oct 30 22:29:44.057590 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:44.057557    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-582593-748c54b589-7kjl4"
Oct 30 22:29:45.855550 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:45.855491    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-3cc1b3-openshift-default-744b6b9d7f-lx8n5"
Oct 30 22:29:45.955327 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:45.955297    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b6efa36-7840-418b-be4a-723ca4407a91-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b6efa36-7840-418b-be4a-723ca4407a91-workload-socket\") pod \"4b6efa36-7840-418b-be4a-723ca4407a91\" (UID: \"4b6efa36-7840-418b-be4a-723ca4407a91\") "
Oct 30 22:29:45.955538 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:45.955380    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b6efa36-7840-418b-be4a-723ca4407a91-istio-token\" (UniqueName: \"kubernetes.io/projected/4b6efa36-7840-418b-be4a-723ca4407a91-istio-token\") pod \"4b6efa36-7840-418b-be4a-723ca4407a91\" (UID: \"4b6efa36-7840-418b-be4a-723ca4407a91\") "
Oct 30 22:29:45.955538 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:45.955426    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b6efa36-7840-418b-be4a-723ca4407a91-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b6efa36-7840-418b-be4a-723ca4407a91-istio-data\") pod \"4b6efa36-7840-418b-be4a-723ca4407a91\" (UID: \"4b6efa36-7840-418b-be4a-723ca4407a91\") "
Oct 30 22:29:45.955538 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:45.955497    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b6efa36-7840-418b-be4a-723ca4407a91-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b6efa36-7840-418b-be4a-723ca4407a91-istio-envoy\") pod \"4b6efa36-7840-418b-be4a-723ca4407a91\" (UID: \"4b6efa36-7840-418b-be4a-723ca4407a91\") "
Oct 30 22:29:45.955538 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:45.955521    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b6efa36-7840-418b-be4a-723ca4407a91-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b6efa36-7840-418b-be4a-723ca4407a91-credential-socket\") pod \"4b6efa36-7840-418b-be4a-723ca4407a91\" (UID: \"4b6efa36-7840-418b-be4a-723ca4407a91\") "
Oct 30 22:29:45.955754 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:45.955545    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b6efa36-7840-418b-be4a-723ca4407a91-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b6efa36-7840-418b-be4a-723ca4407a91-workload-certs\") pod \"4b6efa36-7840-418b-be4a-723ca4407a91\" (UID: \"4b6efa36-7840-418b-be4a-723ca4407a91\") "
Oct 30 22:29:45.955754 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:45.955575    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4b6efa36-7840-418b-be4a-723ca4407a91-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b6efa36-7840-418b-be4a-723ca4407a91-istio-podinfo\") pod \"4b6efa36-7840-418b-be4a-723ca4407a91\" (UID: \"4b6efa36-7840-418b-be4a-723ca4407a91\") "
Oct 30 22:29:45.955754 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:45.955637    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b6efa36-7840-418b-be4a-723ca4407a91-workload-socket" pod "4b6efa36-7840-418b-be4a-723ca4407a91" (UID: "4b6efa36-7840-418b-be4a-723ca4407a91"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:45.955754 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:45.955716    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b6efa36-7840-418b-be4a-723ca4407a91-istio-data" pod "4b6efa36-7840-418b-be4a-723ca4407a91" (UID: "4b6efa36-7840-418b-be4a-723ca4407a91"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:45.955987 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:45.955775    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b6efa36-7840-418b-be4a-723ca4407a91-kube-api-access-8xwgg\" (UniqueName: \"kubernetes.io/projected/4b6efa36-7840-418b-be4a-723ca4407a91-kube-api-access-8xwgg\") pod \"4b6efa36-7840-418b-be4a-723ca4407a91\" (UID: \"4b6efa36-7840-418b-be4a-723ca4407a91\") "
Oct 30 22:29:45.955987 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:45.955812    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b6efa36-7840-418b-be4a-723ca4407a91-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b6efa36-7840-418b-be4a-723ca4407a91-istiod-ca-cert\") pod \"4b6efa36-7840-418b-be4a-723ca4407a91\" (UID: \"4b6efa36-7840-418b-be4a-723ca4407a91\") "
Oct 30 22:29:45.955987 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:45.955843    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b6efa36-7840-418b-be4a-723ca4407a91-credential-socket" pod "4b6efa36-7840-418b-be4a-723ca4407a91" (UID: "4b6efa36-7840-418b-be4a-723ca4407a91"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:45.955987 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:45.955863    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b6efa36-7840-418b-be4a-723ca4407a91-workload-certs" pod "4b6efa36-7840-418b-be4a-723ca4407a91" (UID: "4b6efa36-7840-418b-be4a-723ca4407a91"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:45.956202 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:45.956169    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b6efa36-7840-418b-be4a-723ca4407a91-istiod-ca-cert" pod "4b6efa36-7840-418b-be4a-723ca4407a91" (UID: "4b6efa36-7840-418b-be4a-723ca4407a91"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:29:45.956511 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:45.956486    5678 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b6efa36-7840-418b-be4a-723ca4407a91-istio-data\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:45.956631 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:45.956516    5678 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b6efa36-7840-418b-be4a-723ca4407a91-credential-socket\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:45.956631 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:45.956532    5678 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b6efa36-7840-418b-be4a-723ca4407a91-workload-certs\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:45.956631 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:45.956546    5678 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b6efa36-7840-418b-be4a-723ca4407a91-istiod-ca-cert\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:45.956631 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:45.956561    5678 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b6efa36-7840-418b-be4a-723ca4407a91-workload-socket\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:45.957751 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:45.957727    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4b6efa36-7840-418b-be4a-723ca4407a91-istio-podinfo" pod "4b6efa36-7840-418b-be4a-723ca4407a91" (UID: "4b6efa36-7840-418b-be4a-723ca4407a91"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:29:45.957855 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:45.957820    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b6efa36-7840-418b-be4a-723ca4407a91-istio-token" pod "4b6efa36-7840-418b-be4a-723ca4407a91" (UID: "4b6efa36-7840-418b-be4a-723ca4407a91"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:29:45.957926 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:45.957851    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b6efa36-7840-418b-be4a-723ca4407a91-istio-envoy" pod "4b6efa36-7840-418b-be4a-723ca4407a91" (UID: "4b6efa36-7840-418b-be4a-723ca4407a91"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:45.957980 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:45.957938    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b6efa36-7840-418b-be4a-723ca4407a91-kube-api-access-8xwgg" pod "4b6efa36-7840-418b-be4a-723ca4407a91" (UID: "4b6efa36-7840-418b-be4a-723ca4407a91"). InnerVolumeSpecName "kube-api-access-8xwgg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:29:46.058157 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:46.058127    5678 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b6efa36-7840-418b-be4a-723ca4407a91-istio-token\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:46.058157 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:46.058156    5678 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b6efa36-7840-418b-be4a-723ca4407a91-istio-envoy\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:46.058157 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:46.058171    5678 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b6efa36-7840-418b-be4a-723ca4407a91-istio-podinfo\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:46.058388 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:46.058185    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xwgg\" (UniqueName: \"kubernetes.io/projected/4b6efa36-7840-418b-be4a-723ca4407a91-kube-api-access-8xwgg\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:46.137905 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:46.137879    5678 generic.go:356] "Generic (PLEG): container finished" podID="4b6efa36-7840-418b-be4a-723ca4407a91" containerID="d21d441a96db7f14edbbc29043f50740221659824026888da3bcaf666bc476de" exitCode=0
Oct 30 22:29:46.138060 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:46.137919    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-3cc1b3-openshift-default-744b6b9d7f-lx8n5" event={"ID":"4b6efa36-7840-418b-be4a-723ca4407a91","Type":"ContainerDied","Data":"d21d441a96db7f14edbbc29043f50740221659824026888da3bcaf666bc476de"}
Oct 30 22:29:46.138060 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:46.137979    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-3cc1b3-openshift-default-744b6b9d7f-lx8n5" event={"ID":"4b6efa36-7840-418b-be4a-723ca4407a91","Type":"ContainerDied","Data":"fb6aa72538d513818b553307983046e93fe1c0731ac3a321ebdb8cceb3826464"}
Oct 30 22:29:46.138060 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:46.138007    5678 scope.go:122] "RemoveContainer" containerID="d21d441a96db7f14edbbc29043f50740221659824026888da3bcaf666bc476de"
Oct 30 22:29:46.138170 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:46.137933    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-3cc1b3-openshift-default-744b6b9d7f-lx8n5"
Oct 30 22:29:46.148887 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:46.148865    5678 scope.go:122] "RemoveContainer" containerID="0f413616c94f35de7224875324a6cc2aa3bc8255d48c60b79f78ca0fafddd51d"
Oct 30 22:29:46.157722 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:46.157703    5678 scope.go:122] "RemoveContainer" containerID="d21d441a96db7f14edbbc29043f50740221659824026888da3bcaf666bc476de"
Oct 30 22:29:46.157986 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:29:46.157967    5678 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d21d441a96db7f14edbbc29043f50740221659824026888da3bcaf666bc476de\": container with ID starting with d21d441a96db7f14edbbc29043f50740221659824026888da3bcaf666bc476de not found: ID does not exist" containerID="d21d441a96db7f14edbbc29043f50740221659824026888da3bcaf666bc476de"
Oct 30 22:29:46.158063 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:46.157996    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d21d441a96db7f14edbbc29043f50740221659824026888da3bcaf666bc476de"} err="failed to get container status \"d21d441a96db7f14edbbc29043f50740221659824026888da3bcaf666bc476de\": rpc error: code = NotFound desc = could not find container \"d21d441a96db7f14edbbc29043f50740221659824026888da3bcaf666bc476de\": container with ID starting with d21d441a96db7f14edbbc29043f50740221659824026888da3bcaf666bc476de not found: ID does not exist"
Oct 30 22:29:46.158063 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:46.158021    5678 scope.go:122] "RemoveContainer" containerID="0f413616c94f35de7224875324a6cc2aa3bc8255d48c60b79f78ca0fafddd51d"
Oct 30 22:29:46.158265 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:29:46.158247    5678 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f413616c94f35de7224875324a6cc2aa3bc8255d48c60b79f78ca0fafddd51d\": container with ID starting with 0f413616c94f35de7224875324a6cc2aa3bc8255d48c60b79f78ca0fafddd51d not found: ID does not exist" containerID="0f413616c94f35de7224875324a6cc2aa3bc8255d48c60b79f78ca0fafddd51d"
Oct 30 22:29:46.158306 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:46.158272    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f413616c94f35de7224875324a6cc2aa3bc8255d48c60b79f78ca0fafddd51d"} err="failed to get container status \"0f413616c94f35de7224875324a6cc2aa3bc8255d48c60b79f78ca0fafddd51d\": rpc error: code = NotFound desc = could not find container \"0f413616c94f35de7224875324a6cc2aa3bc8255d48c60b79f78ca0fafddd51d\": container with ID starting with 0f413616c94f35de7224875324a6cc2aa3bc8255d48c60b79f78ca0fafddd51d not found: ID does not exist"
Oct 30 22:29:46.295919 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:46.295888    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-3cc1b3-openshift-default-744b6b9d7f-lx8n5"]
Oct 30 22:29:46.300474 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:46.300448    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-3cc1b3-openshift-default-744b6b9d7f-lx8n5"]
Oct 30 22:29:46.524767 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:46.524698    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b6efa36-7840-418b-be4a-723ca4407a91" path="/var/lib/kubelet/pods/4b6efa36-7840-418b-be4a-723ca4407a91/volumes"
Oct 30 22:29:54.553104 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:54.553063    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cdbcf"]
Oct 30 22:29:54.553854 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:54.553283    5678 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cdbcf" podUID="4618a35f-6775-4be2-ab02-e5c9940e13b0" containerName="istio-proxy" containerID="cri-o://0f982004d8444fe33ae8a1407a7f808eabc15c117e0d2bacfa68885ce6d11d97" gracePeriod=30
Oct 30 22:29:56.058586 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:56.058549    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-zl688"]
Oct 30 22:29:56.059077 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:56.058862    5678 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-zl688" podUID="c6373802-bc1e-4606-b946-4ae9753c560a" containerName="istio-proxy" containerID="cri-o://fad35cf21cdba6b83a349e4852266f7563805393e18a06784f6b76505d00090d" gracePeriod=30
Oct 30 22:29:57.360340 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:57.360304    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-jps22"]
Oct 30 22:29:57.360823 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:57.360530    5678 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-jps22" podUID="d167328f-3684-41d7-9407-5ceec39dd03f" containerName="istio-proxy" containerID="cri-o://f37eeb0872bd53abc956a7e3019a5df01eb46cfff5dbaa91d136ab47473f9105" gracePeriod=30
Oct 30 22:29:57.644450 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:57.644417    5678 patch_prober.go:29] interesting pod/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cdbcf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:29:57.644618 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:57.644491    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cdbcf" podUID="4618a35f-6775-4be2-ab02-e5c9940e13b0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:29:58.049707 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:58.049627    5678 patch_prober.go:29] interesting pod/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-zl688 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:29:58.049882 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:58.049700    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-zl688" podUID="c6373802-bc1e-4606-b946-4ae9753c560a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:29:59.825844 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:59.825780    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cdbcf"
Oct 30 22:29:59.968328 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:59.968297    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4618a35f-6775-4be2-ab02-e5c9940e13b0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4618a35f-6775-4be2-ab02-e5c9940e13b0-istio-envoy\") pod \"4618a35f-6775-4be2-ab02-e5c9940e13b0\" (UID: \"4618a35f-6775-4be2-ab02-e5c9940e13b0\") "
Oct 30 22:29:59.968328 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:59.968335    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4618a35f-6775-4be2-ab02-e5c9940e13b0-istio-token\" (UniqueName: \"kubernetes.io/projected/4618a35f-6775-4be2-ab02-e5c9940e13b0-istio-token\") pod \"4618a35f-6775-4be2-ab02-e5c9940e13b0\" (UID: \"4618a35f-6775-4be2-ab02-e5c9940e13b0\") "
Oct 30 22:29:59.968598 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:59.968361    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4618a35f-6775-4be2-ab02-e5c9940e13b0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4618a35f-6775-4be2-ab02-e5c9940e13b0-istio-data\") pod \"4618a35f-6775-4be2-ab02-e5c9940e13b0\" (UID: \"4618a35f-6775-4be2-ab02-e5c9940e13b0\") "
Oct 30 22:29:59.968598 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:59.968388    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4618a35f-6775-4be2-ab02-e5c9940e13b0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4618a35f-6775-4be2-ab02-e5c9940e13b0-workload-certs\") pod \"4618a35f-6775-4be2-ab02-e5c9940e13b0\" (UID: \"4618a35f-6775-4be2-ab02-e5c9940e13b0\") "
Oct 30 22:29:59.968598 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:59.968460    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4618a35f-6775-4be2-ab02-e5c9940e13b0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4618a35f-6775-4be2-ab02-e5c9940e13b0-istio-podinfo\") pod \"4618a35f-6775-4be2-ab02-e5c9940e13b0\" (UID: \"4618a35f-6775-4be2-ab02-e5c9940e13b0\") "
Oct 30 22:29:59.968598 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:59.968489    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4618a35f-6775-4be2-ab02-e5c9940e13b0-kube-api-access-9c9gs\" (UniqueName: \"kubernetes.io/projected/4618a35f-6775-4be2-ab02-e5c9940e13b0-kube-api-access-9c9gs\") pod \"4618a35f-6775-4be2-ab02-e5c9940e13b0\" (UID: \"4618a35f-6775-4be2-ab02-e5c9940e13b0\") "
Oct 30 22:29:59.968813 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:59.968597    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4618a35f-6775-4be2-ab02-e5c9940e13b0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4618a35f-6775-4be2-ab02-e5c9940e13b0-credential-socket\") pod \"4618a35f-6775-4be2-ab02-e5c9940e13b0\" (UID: \"4618a35f-6775-4be2-ab02-e5c9940e13b0\") "
Oct 30 22:29:59.968813 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:59.968621    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4618a35f-6775-4be2-ab02-e5c9940e13b0-workload-certs" pod "4618a35f-6775-4be2-ab02-e5c9940e13b0" (UID: "4618a35f-6775-4be2-ab02-e5c9940e13b0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:59.968813 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:59.968645    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4618a35f-6775-4be2-ab02-e5c9940e13b0-istio-data" pod "4618a35f-6775-4be2-ab02-e5c9940e13b0" (UID: "4618a35f-6775-4be2-ab02-e5c9940e13b0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:59.968813 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:59.968704    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4618a35f-6775-4be2-ab02-e5c9940e13b0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4618a35f-6775-4be2-ab02-e5c9940e13b0-istiod-ca-cert\") pod \"4618a35f-6775-4be2-ab02-e5c9940e13b0\" (UID: \"4618a35f-6775-4be2-ab02-e5c9940e13b0\") "
Oct 30 22:29:59.969019 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:59.968879    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4618a35f-6775-4be2-ab02-e5c9940e13b0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4618a35f-6775-4be2-ab02-e5c9940e13b0-workload-socket\") pod \"4618a35f-6775-4be2-ab02-e5c9940e13b0\" (UID: \"4618a35f-6775-4be2-ab02-e5c9940e13b0\") "
Oct 30 22:29:59.969081 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:59.969021    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4618a35f-6775-4be2-ab02-e5c9940e13b0-credential-socket" pod "4618a35f-6775-4be2-ab02-e5c9940e13b0" (UID: "4618a35f-6775-4be2-ab02-e5c9940e13b0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:59.969139 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:59.969065    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4618a35f-6775-4be2-ab02-e5c9940e13b0-istiod-ca-cert" pod "4618a35f-6775-4be2-ab02-e5c9940e13b0" (UID: "4618a35f-6775-4be2-ab02-e5c9940e13b0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:29:59.969338 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:59.969313    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4618a35f-6775-4be2-ab02-e5c9940e13b0-workload-socket" pod "4618a35f-6775-4be2-ab02-e5c9940e13b0" (UID: "4618a35f-6775-4be2-ab02-e5c9940e13b0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:59.969662 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:59.969637    5678 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4618a35f-6775-4be2-ab02-e5c9940e13b0-workload-socket\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:59.969737 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:59.969667    5678 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4618a35f-6775-4be2-ab02-e5c9940e13b0-istio-data\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:59.969737 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:59.969683    5678 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4618a35f-6775-4be2-ab02-e5c9940e13b0-workload-certs\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:59.969737 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:59.969698    5678 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4618a35f-6775-4be2-ab02-e5c9940e13b0-credential-socket\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:59.969737 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:59.969712    5678 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4618a35f-6775-4be2-ab02-e5c9940e13b0-istiod-ca-cert\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:59.970582 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:59.970554    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4618a35f-6775-4be2-ab02-e5c9940e13b0-istio-podinfo" pod "4618a35f-6775-4be2-ab02-e5c9940e13b0" (UID: "4618a35f-6775-4be2-ab02-e5c9940e13b0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:29:59.970719 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:59.970698    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4618a35f-6775-4be2-ab02-e5c9940e13b0-istio-token" pod "4618a35f-6775-4be2-ab02-e5c9940e13b0" (UID: "4618a35f-6775-4be2-ab02-e5c9940e13b0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:29:59.970777 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:59.970766    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4618a35f-6775-4be2-ab02-e5c9940e13b0-istio-envoy" pod "4618a35f-6775-4be2-ab02-e5c9940e13b0" (UID: "4618a35f-6775-4be2-ab02-e5c9940e13b0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:59.970818 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:29:59.970771    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4618a35f-6775-4be2-ab02-e5c9940e13b0-kube-api-access-9c9gs" pod "4618a35f-6775-4be2-ab02-e5c9940e13b0" (UID: "4618a35f-6775-4be2-ab02-e5c9940e13b0"). InnerVolumeSpecName "kube-api-access-9c9gs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:00.071282 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:00.071250    5678 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4618a35f-6775-4be2-ab02-e5c9940e13b0-istio-envoy\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:00.071282 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:00.071275    5678 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4618a35f-6775-4be2-ab02-e5c9940e13b0-istio-token\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:00.071282 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:00.071285    5678 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4618a35f-6775-4be2-ab02-e5c9940e13b0-istio-podinfo\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:00.071570 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:00.071294    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9c9gs\" (UniqueName: \"kubernetes.io/projected/4618a35f-6775-4be2-ab02-e5c9940e13b0-kube-api-access-9c9gs\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:00.184504 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:00.184460    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c42c0a-29889990-z4dwc"]
Oct 30 22:30:00.185268 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:00.185241    5678 cpu_manager.go:414] "containerMap: removing container" podUID="4b6efa36-7840-418b-be4a-723ca4407a91" containerName="istio-proxy"
Oct 30 22:30:00.185268 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:00.185264    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b6efa36-7840-418b-be4a-723ca4407a91" containerName="istio-proxy"
Oct 30 22:30:00.185268 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:00.185275    5678 cpu_manager.go:414] "containerMap: removing container" podUID="4618a35f-6775-4be2-ab02-e5c9940e13b0" containerName="istio-proxy"
Oct 30 22:30:00.185567 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:00.185281    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4618a35f-6775-4be2-ab02-e5c9940e13b0" containerName="istio-proxy"
Oct 30 22:30:00.185567 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:00.185297    5678 cpu_manager.go:414] "containerMap: removing container" podUID="4b6efa36-7840-418b-be4a-723ca4407a91" containerName="istio-proxy"
Oct 30 22:30:00.185567 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:00.185302    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b6efa36-7840-418b-be4a-723ca4407a91" containerName="istio-proxy"
Oct 30 22:30:00.185567 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:00.185311    5678 cpu_manager.go:414] "containerMap: removing container" podUID="4618a35f-6775-4be2-ab02-e5c9940e13b0" containerName="istio-proxy"
Oct 30 22:30:00.185567 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:00.185316    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4618a35f-6775-4be2-ab02-e5c9940e13b0" containerName="istio-proxy"
Oct 30 22:30:00.185567 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:00.185433    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b6efa36-7840-418b-be4a-723ca4407a91" containerName="istio-proxy"
Oct 30 22:30:00.185567 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:00.185442    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="4618a35f-6775-4be2-ab02-e5c9940e13b0" containerName="istio-proxy"
Oct 30 22:30:00.185567 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:00.185451    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b6efa36-7840-418b-be4a-723ca4407a91" containerName="istio-proxy"
Oct 30 22:30:00.189563 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:00.189545    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c42c0a-29889990-z4dwc"
Oct 30 22:30:00.208436 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:00.208393    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-99304b-29889990-wn2vq"]
Oct 30 22:30:00.209392 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:00.209360    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="4618a35f-6775-4be2-ab02-e5c9940e13b0" containerName="istio-proxy"
Oct 30 22:30:00.212938 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:00.212921    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-99304b-29889990-wn2vq"
Oct 30 22:30:00.228131 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:00.228105    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb0b93-29889990-dldtk"]
Oct 30 22:30:00.232216 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:00.232191    5678 generic.go:356] "Generic (PLEG): container finished" podID="4618a35f-6775-4be2-ab02-e5c9940e13b0" containerID="0f982004d8444fe33ae8a1407a7f808eabc15c117e0d2bacfa68885ce6d11d97" exitCode=0
Oct 30 22:30:00.232342 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:00.232313    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cdbcf"
Oct 30 22:30:00.232640 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:00.232617    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cdbcf" event={"ID":"4618a35f-6775-4be2-ab02-e5c9940e13b0","Type":"ContainerDied","Data":"0f982004d8444fe33ae8a1407a7f808eabc15c117e0d2bacfa68885ce6d11d97"}
Oct 30 22:30:00.232876 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:00.232859    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cdbcf" event={"ID":"4618a35f-6775-4be2-ab02-e5c9940e13b0","Type":"ContainerDied","Data":"51b0b5b7343ef4a4f1579a876a08bfb83870d8ff522ddf526d14314b84cdc43c"}
Oct 30 22:30:00.233032 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:00.232984    5678 scope.go:122] "RemoveContainer" containerID="0f982004d8444fe33ae8a1407a7f808eabc15c117e0d2bacfa68885ce6d11d97"
Oct 30 22:30:00.233948 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:00.233927    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb0b93-29889990-dldtk"
Oct 30 22:30:00.250977 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:00.250945    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-779f32-29889990-m7sk5"]
Oct 30 22:30:00.264264 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:00.264240    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bc714d-29889990-cxr7b"]
Oct 30 22:30:00.264463 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:00.264444    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-779f32-29889990-m7sk5"
Oct 30 22:30:00.266812 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:00.266792    5678 scope.go:122] "RemoveContainer" containerID="656ee0b1af412dbc9cc40e18f123f9e2dc69a6ac2770b6536e90afe20e196612"
Oct 30 22:30:00.271294 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:00.271237    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bc714d-29889990-cxr7b"
Oct 30 22:30:00.279131 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:00.279106    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c42c0a-29889990-z4dwc"]
Oct 30 22:30:00.292543 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:00.292088    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-99304b-29889990-wn2vq"]
Oct 30 22:30:00.304725 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:00.298636    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5rvcj\""
Oct 30 22:30:00.304725 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:00.298719    5678 scope.go:122] "RemoveContainer" containerID="0f982004d8444fe33ae8a1407a7f808eabc15c117e0d2bacfa68885ce6d11d97"
Oct 30 22:30:00.304725 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:30:00.301327    5678 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f982004d8444fe33ae8a1407a7f808eabc15c117e0d2bacfa68885ce6d11d97\": container with ID starting with 0f982004d8444fe33ae8a1407a7f808eabc15c117e0d2bacfa68885ce6d11d97 not found: ID does not exist" containerID="0f982004d8444fe33ae8a1407a7f808eabc15c117e0d2bacfa68885ce6d11d97"
Oct 30 22:30:00.304725 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:00.301367    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f982004d8444fe33ae8a1407a7f808eabc15c117e0d2bacfa68885ce6d11d97"} err="failed to get container status \"0f982004d8444fe33ae8a1407a7f808eabc15c117e0d2bacfa68885ce6d11d97\": rpc error: code = NotFound desc = could not find container \"0f982004d8444fe33ae8a1407a7f808eabc15c117e0d2bacfa68885ce6d11d97\": container with ID starting with 0f982004d8444fe33ae8a1407a7f808eabc15c117e0d2bacfa68885ce6d11d97 not found: ID does not exist"
Oct 30 22:30:00.304725 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:00.301393    5678 scope.go:122] "RemoveContainer" containerID="656ee0b1af412dbc9cc40e18f123f9e2dc69a6ac2770b6536e90afe20e196612"
Oct 30 22:30:00.304725 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:30:00.301780    5678 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"656ee0b1af412dbc9cc40e18f123f9e2dc69a6ac2770b6536e90afe20e196612\": container with ID starting with 656ee0b1af412dbc9cc40e18f123f9e2dc69a6ac2770b6536e90afe20e196612 not found: ID does not exist" containerID="656ee0b1af412dbc9cc40e18f123f9e2dc69a6ac2770b6536e90afe20e196612"
Oct 30 22:30:00.304725 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:00.301813    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"656ee0b1af412dbc9cc40e18f123f9e2dc69a6ac2770b6536e90afe20e196612"} err="failed to get container status \"656ee0b1af412dbc9cc40e18f123f9e2dc69a6ac2770b6536e90afe20e196612\": rpc error: code = NotFound desc = could not find container \"656ee0b1af412dbc9cc40e18f123f9e2dc69a6ac2770b6536e90afe20e196612\": container with ID starting with 656ee0b1af412dbc9cc40e18f123f9e2dc69a6ac2770b6536e90afe20e196612 not found: ID does not exist"
Oct 30 22:30:00.306437 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:00.306351    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb0b93-29889990-dldtk"]
Oct 30 22:30:00.327469 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:00.322200    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-779f32-29889990-m7sk5"]
Oct 30 22:30:00.331411 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:00.329947    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bc714d-29889990-cxr7b"]
Oct 30 22:30:00.376503 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:00.376458    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcljh\" (UniqueName: \"kubernetes.io/projected/d0b1525a-699c-4121-94dd-f47d00b08ae7-kube-api-access-mcljh\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-c42c0a-29889990-z4dwc\" (UID: \"d0b1525a-699c-4121-94dd-f47d00b08ae7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c42c0a-29889990-z4dwc"
Oct 30 22:30:00.376681 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:00.376599    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0b1525a-699c-4121-94dd-f47d00b08ae7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-c42c0a-29889990-z4dwc\" (UID: \"d0b1525a-699c-4121-94dd-f47d00b08ae7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c42c0a-29889990-z4dwc"
Oct 30 22:30:00.376747 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:00.376683    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjf4w\" (UniqueName: \"kubernetes.io/projected/a94efb34-dd76-4be4-9872-947273ced7c0-kube-api-access-qjf4w\") pod \"maas-api-key-cleanup-e2e-worker-w2-bc714d-29889990-cxr7b\" (UID: \"a94efb34-dd76-4be4-9872-947273ced7c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bc714d-29889990-cxr7b"
Oct 30 22:30:00.376857 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:00.376837    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d886be7a-7208-446a-a3aa-6c478d277578-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-99304b-29889990-wn2vq\" (UID: \"d886be7a-7208-446a-a3aa-6c478d277578\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-99304b-29889990-wn2vq"
Oct 30 22:30:00.376920 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:00.376873    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8663b2d-a48a-49a1-a788-7de17d54e62b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-fb0b93-29889990-dldtk\" (UID: \"e8663b2d-a48a-49a1-a788-7de17d54e62b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb0b93-29889990-dldtk"
Oct 30 22:30:00.377103 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:00.377082    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pqjz\" (UniqueName: \"kubernetes.io/projected/ddca5ef0-e1ab-493c-a3c8-a15e17d91d70-kube-api-access-8pqjz\") pod \"maas-api-key-cleanup-e2e-worker-w0-779f32-29889990-m7sk5\" (UID: \"ddca5ef0-e1ab-493c-a3c8-a15e17d91d70\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-779f32-29889990-m7sk5"
Oct 30 22:30:00.377177 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:00.377160    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjl9x\" (UniqueName: \"kubernetes.io/projected/d886be7a-7208-446a-a3aa-6c478d277578-kube-api-access-wjl9x\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-99304b-29889990-wn2vq\" (UID: \"d886be7a-7208-446a-a3aa-6c478d277578\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-99304b-29889990-wn2vq"
Oct 30 22:30:00.377231 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:00.377193    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxkzt\" (UniqueName: \"kubernetes.io/projected/e8663b2d-a48a-49a1-a788-7de17d54e62b-kube-api-access-zxkzt\") pod \"maas-api-key-cleanup-e2e-worker-w6-fb0b93-29889990-dldtk\" (UID: \"e8663b2d-a48a-49a1-a788-7de17d54e62b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb0b93-29889990-dldtk"
Oct 30 22:30:00.377384 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:00.377312    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddca5ef0-e1ab-493c-a3c8-a15e17d91d70-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-779f32-29889990-m7sk5\" (UID: \"ddca5ef0-e1ab-493c-a3c8-a15e17d91d70\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-779f32-29889990-m7sk5"
Oct 30 22:30:00.377384 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:00.377342    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a94efb34-dd76-4be4-9872-947273ced7c0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-bc714d-29889990-cxr7b\" (UID: \"a94efb34-dd76-4be4-9872-947273ced7c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bc714d-29889990-cxr7b"
Oct 30 22:30:00.478151 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:00.478066    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcljh\" (UniqueName: \"kubernetes.io/projected/d0b1525a-699c-4121-94dd-f47d00b08ae7-kube-api-access-mcljh\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-c42c0a-29889990-z4dwc\" (UID: \"d0b1525a-699c-4121-94dd-f47d00b08ae7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c42c0a-29889990-z4dwc"
Oct 30 22:30:00.478151 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:00.478119    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0b1525a-699c-4121-94dd-f47d00b08ae7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-c42c0a-29889990-z4dwc\" (UID: \"d0b1525a-699c-4121-94dd-f47d00b08ae7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c42c0a-29889990-z4dwc"
Oct 30 22:30:00.478436 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:00.478157    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjf4w\" (UniqueName: \"kubernetes.io/projected/a94efb34-dd76-4be4-9872-947273ced7c0-kube-api-access-qjf4w\") pod \"maas-api-key-cleanup-e2e-worker-w2-bc714d-29889990-cxr7b\" (UID: \"a94efb34-dd76-4be4-9872-947273ced7c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bc714d-29889990-cxr7b"
Oct 30 22:30:00.478436 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:00.478201    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d886be7a-7208-446a-a3aa-6c478d277578-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-99304b-29889990-wn2vq\" (UID: \"d886be7a-7208-446a-a3aa-6c478d277578\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-99304b-29889990-wn2vq"
Oct 30 22:30:00.478436 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:00.478230    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8663b2d-a48a-49a1-a788-7de17d54e62b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-fb0b93-29889990-dldtk\" (UID: \"e8663b2d-a48a-49a1-a788-7de17d54e62b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb0b93-29889990-dldtk"
Oct 30 22:30:00.478436 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:00.478364    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pqjz\" (UniqueName: \"kubernetes.io/projected/ddca5ef0-e1ab-493c-a3c8-a15e17d91d70-kube-api-access-8pqjz\") pod \"maas-api-key-cleanup-e2e-worker-w0-779f32-29889990-m7sk5\" (UID: \"ddca5ef0-e1ab-493c-a3c8-a15e17d91d70\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-779f32-29889990-m7sk5"
Oct 30 22:30:00.478689 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:00.478444    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjl9x\" (UniqueName: \"kubernetes.io/projected/d886be7a-7208-446a-a3aa-6c478d277578-kube-api-access-wjl9x\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-99304b-29889990-wn2vq\" (UID: \"d886be7a-7208-446a-a3aa-6c478d277578\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-99304b-29889990-wn2vq"
Oct 30 22:30:00.478689 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:00.478473    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxkzt\" (UniqueName: \"kubernetes.io/projected/e8663b2d-a48a-49a1-a788-7de17d54e62b-kube-api-access-zxkzt\") pod \"maas-api-key-cleanup-e2e-worker-w6-fb0b93-29889990-dldtk\" (UID: \"e8663b2d-a48a-49a1-a788-7de17d54e62b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb0b93-29889990-dldtk"
Oct 30 22:30:00.478689 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:00.478568    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddca5ef0-e1ab-493c-a3c8-a15e17d91d70-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-779f32-29889990-m7sk5\" (UID: \"ddca5ef0-e1ab-493c-a3c8-a15e17d91d70\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-779f32-29889990-m7sk5"
Oct 30 22:30:00.478689 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:00.478593    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a94efb34-dd76-4be4-9872-947273ced7c0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-bc714d-29889990-cxr7b\" (UID: \"a94efb34-dd76-4be4-9872-947273ced7c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bc714d-29889990-cxr7b"
Oct 30 22:30:00.478929 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:00.478904    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d886be7a-7208-446a-a3aa-6c478d277578-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-99304b-29889990-wn2vq\" (UID: \"d886be7a-7208-446a-a3aa-6c478d277578\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-99304b-29889990-wn2vq"
Oct 30 22:30:00.478993 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:00.478973    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0b1525a-699c-4121-94dd-f47d00b08ae7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-c42c0a-29889990-z4dwc\" (UID: \"d0b1525a-699c-4121-94dd-f47d00b08ae7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c42c0a-29889990-z4dwc"
Oct 30 22:30:00.479069 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:00.479047    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8663b2d-a48a-49a1-a788-7de17d54e62b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-fb0b93-29889990-dldtk\" (UID: \"e8663b2d-a48a-49a1-a788-7de17d54e62b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb0b93-29889990-dldtk"
Oct 30 22:30:00.479289 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:00.479274    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddca5ef0-e1ab-493c-a3c8-a15e17d91d70-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-779f32-29889990-m7sk5\" (UID: \"ddca5ef0-e1ab-493c-a3c8-a15e17d91d70\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-779f32-29889990-m7sk5"
Oct 30 22:30:00.479328 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:00.479272    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a94efb34-dd76-4be4-9872-947273ced7c0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-bc714d-29889990-cxr7b\" (UID: \"a94efb34-dd76-4be4-9872-947273ced7c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bc714d-29889990-cxr7b"
Oct 30 22:30:00.537157 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:00.537116    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxkzt\" (UniqueName: \"kubernetes.io/projected/e8663b2d-a48a-49a1-a788-7de17d54e62b-kube-api-access-zxkzt\") pod \"maas-api-key-cleanup-e2e-worker-w6-fb0b93-29889990-dldtk\" (UID: \"e8663b2d-a48a-49a1-a788-7de17d54e62b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb0b93-29889990-dldtk"
Oct 30 22:30:00.537295 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:00.537189    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcljh\" (UniqueName: \"kubernetes.io/projected/d0b1525a-699c-4121-94dd-f47d00b08ae7-kube-api-access-mcljh\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-c42c0a-29889990-z4dwc\" (UID: \"d0b1525a-699c-4121-94dd-f47d00b08ae7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c42c0a-29889990-z4dwc"
Oct 30 22:30:00.537295 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:00.537261    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjl9x\" (UniqueName: \"kubernetes.io/projected/d886be7a-7208-446a-a3aa-6c478d277578-kube-api-access-wjl9x\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-99304b-29889990-wn2vq\" (UID: \"d886be7a-7208-446a-a3aa-6c478d277578\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-99304b-29889990-wn2vq"
Oct 30 22:30:00.537388 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:00.537338    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pqjz\" (UniqueName: \"kubernetes.io/projected/ddca5ef0-e1ab-493c-a3c8-a15e17d91d70-kube-api-access-8pqjz\") pod \"maas-api-key-cleanup-e2e-worker-w0-779f32-29889990-m7sk5\" (UID: \"ddca5ef0-e1ab-493c-a3c8-a15e17d91d70\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-779f32-29889990-m7sk5"
Oct 30 22:30:00.537507 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:00.537490    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjf4w\" (UniqueName: \"kubernetes.io/projected/a94efb34-dd76-4be4-9872-947273ced7c0-kube-api-access-qjf4w\") pod \"maas-api-key-cleanup-e2e-worker-w2-bc714d-29889990-cxr7b\" (UID: \"a94efb34-dd76-4be4-9872-947273ced7c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bc714d-29889990-cxr7b"
Oct 30 22:30:00.554921 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:00.554895    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb0b93-29889990-dldtk"
Oct 30 22:30:00.607893 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:00.607475    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-779f32-29889990-m7sk5"
Oct 30 22:30:00.613517 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:00.613492    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bc714d-29889990-cxr7b"
Oct 30 22:30:00.803856 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:00.803826    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c42c0a-29889990-z4dwc"
Oct 30 22:30:00.827483 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:00.827224    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-99304b-29889990-wn2vq"
Oct 30 22:30:00.851606 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:00.851342    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cdbcf"]
Oct 30 22:30:00.855577 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:00.855549    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cdbcf"]
Oct 30 22:30:00.944284 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:00.944260    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb0b93-29889990-dldtk"]
Oct 30 22:30:00.947169 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:30:00.947124    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8663b2d_a48a_49a1_a788_7de17d54e62b.slice/crio-f7d84f772202b745cd2018ee09d653e81f1592e196826e5d4e1ff6b881e786b8 WatchSource:0}: Error finding container f7d84f772202b745cd2018ee09d653e81f1592e196826e5d4e1ff6b881e786b8: Status 404 returned error can't find the container with id f7d84f772202b745cd2018ee09d653e81f1592e196826e5d4e1ff6b881e786b8
Oct 30 22:30:01.024531 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:01.024449    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bc714d-29889990-cxr7b"]
Oct 30 22:30:01.031549 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:30:01.031519    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda94efb34_dd76_4be4_9872_947273ced7c0.slice/crio-5ee8d8b3751c6e31840b0ca198c90bca150b0b479853b46b54bc38f912bb6b11 WatchSource:0}: Error finding container 5ee8d8b3751c6e31840b0ca198c90bca150b0b479853b46b54bc38f912bb6b11: Status 404 returned error can't find the container with id 5ee8d8b3751c6e31840b0ca198c90bca150b0b479853b46b54bc38f912bb6b11
Oct 30 22:30:01.047838 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:01.047809    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-779f32-29889990-m7sk5"]
Oct 30 22:30:01.208996 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:01.208959    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c42c0a-29889990-z4dwc"]
Oct 30 22:30:01.260423 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:01.259760    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-99304b-29889990-wn2vq"]
Oct 30 22:30:01.268798 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:01.266587    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c42c0a-29889990-z4dwc" event={"ID":"d0b1525a-699c-4121-94dd-f47d00b08ae7","Type":"ContainerStarted","Data":"0fc0fc770beabc83ca303720f662dd12ed9383d5ae19d080ab045f2611f7d69e"}
Oct 30 22:30:01.269705 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:01.269011    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb0b93-29889990-dldtk" event={"ID":"e8663b2d-a48a-49a1-a788-7de17d54e62b","Type":"ContainerStarted","Data":"ac7337a3690e7b46dc3275ec0634385c8f1d59d68efff629d57ca134052a3551"}
Oct 30 22:30:01.269705 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:01.269058    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb0b93-29889990-dldtk" event={"ID":"e8663b2d-a48a-49a1-a788-7de17d54e62b","Type":"ContainerStarted","Data":"f7d84f772202b745cd2018ee09d653e81f1592e196826e5d4e1ff6b881e786b8"}
Oct 30 22:30:01.272031 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:01.272008    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-779f32-29889990-m7sk5" event={"ID":"ddca5ef0-e1ab-493c-a3c8-a15e17d91d70","Type":"ContainerStarted","Data":"e1008edc0db52708ac4de3ece53ceb7c7b6df4d6918cc9105de6bad50825e7ff"}
Oct 30 22:30:01.273773 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:01.273752    5678 generic.go:356] "Generic (PLEG): container finished" podID="c6373802-bc1e-4606-b946-4ae9753c560a" containerID="fad35cf21cdba6b83a349e4852266f7563805393e18a06784f6b76505d00090d" exitCode=0
Oct 30 22:30:01.273920 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:01.273901    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-zl688" event={"ID":"c6373802-bc1e-4606-b946-4ae9753c560a","Type":"ContainerDied","Data":"fad35cf21cdba6b83a349e4852266f7563805393e18a06784f6b76505d00090d"}
Oct 30 22:30:01.275226 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:01.275174    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bc714d-29889990-cxr7b" event={"ID":"a94efb34-dd76-4be4-9872-947273ced7c0","Type":"ContainerStarted","Data":"5ee8d8b3751c6e31840b0ca198c90bca150b0b479853b46b54bc38f912bb6b11"}
Oct 30 22:30:01.320607 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:01.320582    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-zl688"
Oct 30 22:30:01.383281 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:01.383222    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb0b93-29889990-dldtk" podStartSLOduration=1.38320962 podStartE2EDuration="1.38320962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:01.379964233 +0000 UTC m=+2378.984271786" watchObservedRunningTime="2026-10-30 22:30:01.38320962 +0000 UTC m=+2378.987517170"
Oct 30 22:30:01.392460 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:01.392431    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6373802-bc1e-4606-b946-4ae9753c560a-kube-api-access-5z87k\" (UniqueName: \"kubernetes.io/projected/c6373802-bc1e-4606-b946-4ae9753c560a-kube-api-access-5z87k\") pod \"c6373802-bc1e-4606-b946-4ae9753c560a\" (UID: \"c6373802-bc1e-4606-b946-4ae9753c560a\") "
Oct 30 22:30:01.392712 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:01.392467    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6373802-bc1e-4606-b946-4ae9753c560a-istio-token\" (UniqueName: \"kubernetes.io/projected/c6373802-bc1e-4606-b946-4ae9753c560a-istio-token\") pod \"c6373802-bc1e-4606-b946-4ae9753c560a\" (UID: \"c6373802-bc1e-4606-b946-4ae9753c560a\") "
Oct 30 22:30:01.392712 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:01.392517    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6373802-bc1e-4606-b946-4ae9753c560a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6373802-bc1e-4606-b946-4ae9753c560a-istio-envoy\") pod \"c6373802-bc1e-4606-b946-4ae9753c560a\" (UID: \"c6373802-bc1e-4606-b946-4ae9753c560a\") "
Oct 30 22:30:01.392712 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:01.392538    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6373802-bc1e-4606-b946-4ae9753c560a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6373802-bc1e-4606-b946-4ae9753c560a-credential-socket\") pod \"c6373802-bc1e-4606-b946-4ae9753c560a\" (UID: \"c6373802-bc1e-4606-b946-4ae9753c560a\") "
Oct 30 22:30:01.392712 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:01.392679    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6373802-bc1e-4606-b946-4ae9753c560a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6373802-bc1e-4606-b946-4ae9753c560a-workload-certs\") pod \"c6373802-bc1e-4606-b946-4ae9753c560a\" (UID: \"c6373802-bc1e-4606-b946-4ae9753c560a\") "
Oct 30 22:30:01.392944 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:01.392802    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6373802-bc1e-4606-b946-4ae9753c560a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6373802-bc1e-4606-b946-4ae9753c560a-istiod-ca-cert\") pod \"c6373802-bc1e-4606-b946-4ae9753c560a\" (UID: \"c6373802-bc1e-4606-b946-4ae9753c560a\") "
Oct 30 22:30:01.392944 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:01.392878    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6373802-bc1e-4606-b946-4ae9753c560a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6373802-bc1e-4606-b946-4ae9753c560a-workload-socket\") pod \"c6373802-bc1e-4606-b946-4ae9753c560a\" (UID: \"c6373802-bc1e-4606-b946-4ae9753c560a\") "
Oct 30 22:30:01.392944 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:01.392867    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6373802-bc1e-4606-b946-4ae9753c560a-credential-socket" pod "c6373802-bc1e-4606-b946-4ae9753c560a" (UID: "c6373802-bc1e-4606-b946-4ae9753c560a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:01.392944 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:01.392908    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6373802-bc1e-4606-b946-4ae9753c560a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6373802-bc1e-4606-b946-4ae9753c560a-istio-data\") pod \"c6373802-bc1e-4606-b946-4ae9753c560a\" (UID: \"c6373802-bc1e-4606-b946-4ae9753c560a\") "
Oct 30 22:30:01.392944 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:01.392933    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6373802-bc1e-4606-b946-4ae9753c560a-workload-certs" pod "c6373802-bc1e-4606-b946-4ae9753c560a" (UID: "c6373802-bc1e-4606-b946-4ae9753c560a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:01.393225 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:01.393021    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c6373802-bc1e-4606-b946-4ae9753c560a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6373802-bc1e-4606-b946-4ae9753c560a-istio-podinfo\") pod \"c6373802-bc1e-4606-b946-4ae9753c560a\" (UID: \"c6373802-bc1e-4606-b946-4ae9753c560a\") "
Oct 30 22:30:01.393225 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:01.393071    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6373802-bc1e-4606-b946-4ae9753c560a-workload-socket" pod "c6373802-bc1e-4606-b946-4ae9753c560a" (UID: "c6373802-bc1e-4606-b946-4ae9753c560a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:01.393225 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:01.393177    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6373802-bc1e-4606-b946-4ae9753c560a-istiod-ca-cert" pod "c6373802-bc1e-4606-b946-4ae9753c560a" (UID: "c6373802-bc1e-4606-b946-4ae9753c560a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:01.393225 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:01.393218    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6373802-bc1e-4606-b946-4ae9753c560a-istio-data" pod "c6373802-bc1e-4606-b946-4ae9753c560a" (UID: "c6373802-bc1e-4606-b946-4ae9753c560a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:01.393602 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:01.393574    5678 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6373802-bc1e-4606-b946-4ae9753c560a-istiod-ca-cert\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:01.393602 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:01.393602    5678 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6373802-bc1e-4606-b946-4ae9753c560a-workload-socket\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:01.393800 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:01.393616    5678 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6373802-bc1e-4606-b946-4ae9753c560a-istio-data\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:01.393800 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:01.393625    5678 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6373802-bc1e-4606-b946-4ae9753c560a-credential-socket\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:01.393800 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:01.393633    5678 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6373802-bc1e-4606-b946-4ae9753c560a-workload-certs\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:01.424064 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:01.424019    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c6373802-bc1e-4606-b946-4ae9753c560a-istio-podinfo" pod "c6373802-bc1e-4606-b946-4ae9753c560a" (UID: "c6373802-bc1e-4606-b946-4ae9753c560a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:30:01.424064 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:01.424037    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6373802-bc1e-4606-b946-4ae9753c560a-istio-envoy" pod "c6373802-bc1e-4606-b946-4ae9753c560a" (UID: "c6373802-bc1e-4606-b946-4ae9753c560a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:01.424229 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:01.424102    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6373802-bc1e-4606-b946-4ae9753c560a-istio-token" pod "c6373802-bc1e-4606-b946-4ae9753c560a" (UID: "c6373802-bc1e-4606-b946-4ae9753c560a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:01.424229 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:01.424207    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6373802-bc1e-4606-b946-4ae9753c560a-kube-api-access-5z87k" pod "c6373802-bc1e-4606-b946-4ae9753c560a" (UID: "c6373802-bc1e-4606-b946-4ae9753c560a"). InnerVolumeSpecName "kube-api-access-5z87k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:01.494844 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:01.494815    5678 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6373802-bc1e-4606-b946-4ae9753c560a-istio-podinfo\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:01.494844 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:01.494845    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5z87k\" (UniqueName: \"kubernetes.io/projected/c6373802-bc1e-4606-b946-4ae9753c560a-kube-api-access-5z87k\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:01.495090 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:01.494863    5678 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6373802-bc1e-4606-b946-4ae9753c560a-istio-token\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:01.495090 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:01.494877    5678 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6373802-bc1e-4606-b946-4ae9753c560a-istio-envoy\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:02.284458 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:02.284419    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c42c0a-29889990-z4dwc" event={"ID":"d0b1525a-699c-4121-94dd-f47d00b08ae7","Type":"ContainerStarted","Data":"8be3010819cd0b051c4469da99f7d6d6bf862c7452a5c8e974e71e880a2c7f06"}
Oct 30 22:30:02.285847 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:02.285821    5678 generic.go:356] "Generic (PLEG): container finished" podID="e8663b2d-a48a-49a1-a788-7de17d54e62b" containerID="ac7337a3690e7b46dc3275ec0634385c8f1d59d68efff629d57ca134052a3551" exitCode=60
Oct 30 22:30:02.285986 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:02.285903    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb0b93-29889990-dldtk" event={"ID":"e8663b2d-a48a-49a1-a788-7de17d54e62b","Type":"ContainerDied","Data":"ac7337a3690e7b46dc3275ec0634385c8f1d59d68efff629d57ca134052a3551"}
Oct 30 22:30:02.286154 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:02.286137    5678 scope.go:122] "RemoveContainer" containerID="ac7337a3690e7b46dc3275ec0634385c8f1d59d68efff629d57ca134052a3551"
Oct 30 22:30:02.287765 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:02.287666    5678 generic.go:356] "Generic (PLEG): container finished" podID="d886be7a-7208-446a-a3aa-6c478d277578" containerID="03703255ca4b198c0b421f99ae7bf75bde3522756f63e8bdfc28ed6b7c900e62" exitCode=60
Oct 30 22:30:02.287765 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:02.287738    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-99304b-29889990-wn2vq" event={"ID":"d886be7a-7208-446a-a3aa-6c478d277578","Type":"ContainerDied","Data":"03703255ca4b198c0b421f99ae7bf75bde3522756f63e8bdfc28ed6b7c900e62"}
Oct 30 22:30:02.287903 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:02.287782    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-99304b-29889990-wn2vq" event={"ID":"d886be7a-7208-446a-a3aa-6c478d277578","Type":"ContainerStarted","Data":"ac7bb0ddc24427bf3e85d82c2c28338fe7fbb1bb5bc456a58d124cd439276d5f"}
Oct 30 22:30:02.287961 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:02.287953    5678 scope.go:122] "RemoveContainer" containerID="03703255ca4b198c0b421f99ae7bf75bde3522756f63e8bdfc28ed6b7c900e62"
Oct 30 22:30:02.289465 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:02.289435    5678 generic.go:356] "Generic (PLEG): container finished" podID="ddca5ef0-e1ab-493c-a3c8-a15e17d91d70" containerID="aef0e96472e5e744dc0fd8090dc05ba59706057ab8809148f27827ccd43e4dd4" exitCode=60
Oct 30 22:30:02.289538 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:02.289494    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-779f32-29889990-m7sk5" event={"ID":"ddca5ef0-e1ab-493c-a3c8-a15e17d91d70","Type":"ContainerDied","Data":"aef0e96472e5e744dc0fd8090dc05ba59706057ab8809148f27827ccd43e4dd4"}
Oct 30 22:30:02.289735 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:02.289719    5678 scope.go:122] "RemoveContainer" containerID="aef0e96472e5e744dc0fd8090dc05ba59706057ab8809148f27827ccd43e4dd4"
Oct 30 22:30:02.290960 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:02.290949    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-zl688" event={"ID":"c6373802-bc1e-4606-b946-4ae9753c560a","Type":"ContainerDied","Data":"340c4e245eab6f343fe1c3f3833d2bf2f410acf584bfc87e526e634863ccc337"}
Oct 30 22:30:02.291028 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:02.290983    5678 scope.go:122] "RemoveContainer" containerID="fad35cf21cdba6b83a349e4852266f7563805393e18a06784f6b76505d00090d"
Oct 30 22:30:02.291092 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:02.291064    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-zl688"
Oct 30 22:30:02.293140 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:02.293117    5678 generic.go:356] "Generic (PLEG): container finished" podID="a94efb34-dd76-4be4-9872-947273ced7c0" containerID="24f87e2877bac4409a1b79698dffc21362be2fafc4435931e4ea9fad31e155b6" exitCode=60
Oct 30 22:30:02.293245 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:02.293153    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bc714d-29889990-cxr7b" event={"ID":"a94efb34-dd76-4be4-9872-947273ced7c0","Type":"ContainerDied","Data":"24f87e2877bac4409a1b79698dffc21362be2fafc4435931e4ea9fad31e155b6"}
Oct 30 22:30:02.293534 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:02.293507    5678 scope.go:122] "RemoveContainer" containerID="24f87e2877bac4409a1b79698dffc21362be2fafc4435931e4ea9fad31e155b6"
Oct 30 22:30:02.395043 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:02.394990    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c42c0a-29889990-z4dwc" podStartSLOduration=2.3949717440000002 podStartE2EDuration="2.394971744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:02.394330771 +0000 UTC m=+2379.998638325" watchObservedRunningTime="2026-10-30 22:30:02.394971744 +0000 UTC m=+2379.999279296"
Oct 30 22:30:02.525131 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:02.525062    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4618a35f-6775-4be2-ab02-e5c9940e13b0" path="/var/lib/kubelet/pods/4618a35f-6775-4be2-ab02-e5c9940e13b0/volumes"
Oct 30 22:30:02.692934 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:02.692907    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-jps22"
Oct 30 22:30:02.713705 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:02.713681    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d167328f-3684-41d7-9407-5ceec39dd03f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d167328f-3684-41d7-9407-5ceec39dd03f-istio-data\") pod \"d167328f-3684-41d7-9407-5ceec39dd03f\" (UID: \"d167328f-3684-41d7-9407-5ceec39dd03f\") "
Oct 30 22:30:02.713818 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:02.713715    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d167328f-3684-41d7-9407-5ceec39dd03f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d167328f-3684-41d7-9407-5ceec39dd03f-credential-socket\") pod \"d167328f-3684-41d7-9407-5ceec39dd03f\" (UID: \"d167328f-3684-41d7-9407-5ceec39dd03f\") "
Oct 30 22:30:02.713818 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:02.713764    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d167328f-3684-41d7-9407-5ceec39dd03f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d167328f-3684-41d7-9407-5ceec39dd03f-workload-socket\") pod \"d167328f-3684-41d7-9407-5ceec39dd03f\" (UID: \"d167328f-3684-41d7-9407-5ceec39dd03f\") "
Oct 30 22:30:02.713818 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:02.713786    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d167328f-3684-41d7-9407-5ceec39dd03f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d167328f-3684-41d7-9407-5ceec39dd03f-istio-podinfo\") pod \"d167328f-3684-41d7-9407-5ceec39dd03f\" (UID: \"d167328f-3684-41d7-9407-5ceec39dd03f\") "
Oct 30 22:30:02.713993 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:02.713833    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d167328f-3684-41d7-9407-5ceec39dd03f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d167328f-3684-41d7-9407-5ceec39dd03f-istiod-ca-cert\") pod \"d167328f-3684-41d7-9407-5ceec39dd03f\" (UID: \"d167328f-3684-41d7-9407-5ceec39dd03f\") "
Oct 30 22:30:02.713993 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:02.713914    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d167328f-3684-41d7-9407-5ceec39dd03f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d167328f-3684-41d7-9407-5ceec39dd03f-workload-certs\") pod \"d167328f-3684-41d7-9407-5ceec39dd03f\" (UID: \"d167328f-3684-41d7-9407-5ceec39dd03f\") "
Oct 30 22:30:02.713993 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:02.713979    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d167328f-3684-41d7-9407-5ceec39dd03f-istio-data" pod "d167328f-3684-41d7-9407-5ceec39dd03f" (UID: "d167328f-3684-41d7-9407-5ceec39dd03f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:02.714140 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:02.713995    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d167328f-3684-41d7-9407-5ceec39dd03f-kube-api-access-7ncwx\" (UniqueName: \"kubernetes.io/projected/d167328f-3684-41d7-9407-5ceec39dd03f-kube-api-access-7ncwx\") pod \"d167328f-3684-41d7-9407-5ceec39dd03f\" (UID: \"d167328f-3684-41d7-9407-5ceec39dd03f\") "
Oct 30 22:30:02.714140 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:02.714004    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d167328f-3684-41d7-9407-5ceec39dd03f-workload-socket" pod "d167328f-3684-41d7-9407-5ceec39dd03f" (UID: "d167328f-3684-41d7-9407-5ceec39dd03f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:02.714140 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:02.714035    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d167328f-3684-41d7-9407-5ceec39dd03f-istio-token\" (UniqueName: \"kubernetes.io/projected/d167328f-3684-41d7-9407-5ceec39dd03f-istio-token\") pod \"d167328f-3684-41d7-9407-5ceec39dd03f\" (UID: \"d167328f-3684-41d7-9407-5ceec39dd03f\") "
Oct 30 22:30:02.714140 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:02.714108    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d167328f-3684-41d7-9407-5ceec39dd03f-credential-socket" pod "d167328f-3684-41d7-9407-5ceec39dd03f" (UID: "d167328f-3684-41d7-9407-5ceec39dd03f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:02.714344 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:02.714160    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d167328f-3684-41d7-9407-5ceec39dd03f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d167328f-3684-41d7-9407-5ceec39dd03f-istio-envoy\") pod \"d167328f-3684-41d7-9407-5ceec39dd03f\" (UID: \"d167328f-3684-41d7-9407-5ceec39dd03f\") "
Oct 30 22:30:02.714344 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:02.714231    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d167328f-3684-41d7-9407-5ceec39dd03f-workload-certs" pod "d167328f-3684-41d7-9407-5ceec39dd03f" (UID: "d167328f-3684-41d7-9407-5ceec39dd03f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:02.714344 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:02.714292    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d167328f-3684-41d7-9407-5ceec39dd03f-istiod-ca-cert" pod "d167328f-3684-41d7-9407-5ceec39dd03f" (UID: "d167328f-3684-41d7-9407-5ceec39dd03f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:02.714758 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:02.714736    5678 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d167328f-3684-41d7-9407-5ceec39dd03f-istio-data\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:02.714859 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:02.714763    5678 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d167328f-3684-41d7-9407-5ceec39dd03f-credential-socket\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:02.714859 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:02.714781    5678 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d167328f-3684-41d7-9407-5ceec39dd03f-workload-socket\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:02.714859 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:02.714795    5678 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d167328f-3684-41d7-9407-5ceec39dd03f-istiod-ca-cert\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:02.714859 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:02.714813    5678 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d167328f-3684-41d7-9407-5ceec39dd03f-workload-certs\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:02.716328 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:02.716298    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d167328f-3684-41d7-9407-5ceec39dd03f-istio-podinfo" pod "d167328f-3684-41d7-9407-5ceec39dd03f" (UID: "d167328f-3684-41d7-9407-5ceec39dd03f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:30:02.716328 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:02.716309    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d167328f-3684-41d7-9407-5ceec39dd03f-istio-envoy" pod "d167328f-3684-41d7-9407-5ceec39dd03f" (UID: "d167328f-3684-41d7-9407-5ceec39dd03f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:02.716483 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:02.716340    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d167328f-3684-41d7-9407-5ceec39dd03f-istio-token" pod "d167328f-3684-41d7-9407-5ceec39dd03f" (UID: "d167328f-3684-41d7-9407-5ceec39dd03f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:02.716483 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:02.716425    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d167328f-3684-41d7-9407-5ceec39dd03f-kube-api-access-7ncwx" pod "d167328f-3684-41d7-9407-5ceec39dd03f" (UID: "d167328f-3684-41d7-9407-5ceec39dd03f"). InnerVolumeSpecName "kube-api-access-7ncwx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:02.816484 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:02.816384    5678 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d167328f-3684-41d7-9407-5ceec39dd03f-istio-envoy\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:02.816484 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:02.816432    5678 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d167328f-3684-41d7-9407-5ceec39dd03f-istio-podinfo\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:02.816484 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:02.816444    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ncwx\" (UniqueName: \"kubernetes.io/projected/d167328f-3684-41d7-9407-5ceec39dd03f-kube-api-access-7ncwx\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:02.816484 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:02.816455    5678 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d167328f-3684-41d7-9407-5ceec39dd03f-istio-token\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:02.841006 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:02.840978    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-zl688"]
Oct 30 22:30:02.844950 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:02.844922    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-zl688"]
Oct 30 22:30:03.301768 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:03.301729    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb0b93-29889990-dldtk" event={"ID":"e8663b2d-a48a-49a1-a788-7de17d54e62b","Type":"ContainerStarted","Data":"95c0d0ca6b670b8c42ffaf41b1afcbdd5ecf9afbef96e392481596d9c473b274"}
Oct 30 22:30:03.304685 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:03.304653    5678 generic.go:356] "Generic (PLEG): container finished" podID="d886be7a-7208-446a-a3aa-6c478d277578" containerID="bc9012044bb792f977418aa81796e9d46263075ea556fbd1e88c605b548f686d" exitCode=60
Oct 30 22:30:03.304829 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:03.304779    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-99304b-29889990-wn2vq" event={"ID":"d886be7a-7208-446a-a3aa-6c478d277578","Type":"ContainerDied","Data":"bc9012044bb792f977418aa81796e9d46263075ea556fbd1e88c605b548f686d"}
Oct 30 22:30:03.304892 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:03.304857    5678 scope.go:122] "RemoveContainer" containerID="03703255ca4b198c0b421f99ae7bf75bde3522756f63e8bdfc28ed6b7c900e62"
Oct 30 22:30:03.305595 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:03.305478    5678 scope.go:122] "RemoveContainer" containerID="bc9012044bb792f977418aa81796e9d46263075ea556fbd1e88c605b548f686d"
Oct 30 22:30:03.306699 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:30:03.305979    5678 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-99304b-29889990-wn2vq_odh-ai-gateway-infra(d886be7a-7208-446a-a3aa-6c478d277578)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-99304b-29889990-wn2vq" podUID="d886be7a-7208-446a-a3aa-6c478d277578"
Oct 30 22:30:03.311550 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:03.311520    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-779f32-29889990-m7sk5" event={"ID":"ddca5ef0-e1ab-493c-a3c8-a15e17d91d70","Type":"ContainerStarted","Data":"b33b64f81a78bc08d930e752afd8d204d7a9cc383139b21eb0864b5ac3a70922"}
Oct 30 22:30:03.315908 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:03.315883    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bc714d-29889990-cxr7b" event={"ID":"a94efb34-dd76-4be4-9872-947273ced7c0","Type":"ContainerStarted","Data":"02116764a900cfabdc3aa79cbaf8e1c7370de025ac47a3b8d8f72360a4d8c019"}
Oct 30 22:30:03.317334 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:03.317312    5678 generic.go:356] "Generic (PLEG): container finished" podID="d167328f-3684-41d7-9407-5ceec39dd03f" containerID="f37eeb0872bd53abc956a7e3019a5df01eb46cfff5dbaa91d136ab47473f9105" exitCode=0
Oct 30 22:30:03.317425 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:03.317374    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-jps22"
Oct 30 22:30:03.317425 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:03.317412    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-jps22" event={"ID":"d167328f-3684-41d7-9407-5ceec39dd03f","Type":"ContainerDied","Data":"f37eeb0872bd53abc956a7e3019a5df01eb46cfff5dbaa91d136ab47473f9105"}
Oct 30 22:30:03.317546 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:03.317460    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-jps22" event={"ID":"d167328f-3684-41d7-9407-5ceec39dd03f","Type":"ContainerDied","Data":"fbc41e71ada7332d6a73aa54eeef55bd72d6b4e2c300c66d20dba2e295b3d667"}
Oct 30 22:30:03.319141 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:03.319114    5678 generic.go:356] "Generic (PLEG): container finished" podID="d0b1525a-699c-4121-94dd-f47d00b08ae7" containerID="8be3010819cd0b051c4469da99f7d6d6bf862c7452a5c8e974e71e880a2c7f06" exitCode=60
Oct 30 22:30:03.319247 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:03.319183    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c42c0a-29889990-z4dwc" event={"ID":"d0b1525a-699c-4121-94dd-f47d00b08ae7","Type":"ContainerDied","Data":"8be3010819cd0b051c4469da99f7d6d6bf862c7452a5c8e974e71e880a2c7f06"}
Oct 30 22:30:03.319492 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:03.319479    5678 scope.go:122] "RemoveContainer" containerID="8be3010819cd0b051c4469da99f7d6d6bf862c7452a5c8e974e71e880a2c7f06"
Oct 30 22:30:03.324537 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:03.323957    5678 scope.go:122] "RemoveContainer" containerID="f37eeb0872bd53abc956a7e3019a5df01eb46cfff5dbaa91d136ab47473f9105"
Oct 30 22:30:03.433330 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:03.433304    5678 scope.go:122] "RemoveContainer" containerID="f37eeb0872bd53abc956a7e3019a5df01eb46cfff5dbaa91d136ab47473f9105"
Oct 30 22:30:03.433779 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:30:03.433749    5678 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f37eeb0872bd53abc956a7e3019a5df01eb46cfff5dbaa91d136ab47473f9105\": container with ID starting with f37eeb0872bd53abc956a7e3019a5df01eb46cfff5dbaa91d136ab47473f9105 not found: ID does not exist" containerID="f37eeb0872bd53abc956a7e3019a5df01eb46cfff5dbaa91d136ab47473f9105"
Oct 30 22:30:03.433845 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:03.433795    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f37eeb0872bd53abc956a7e3019a5df01eb46cfff5dbaa91d136ab47473f9105"} err="failed to get container status \"f37eeb0872bd53abc956a7e3019a5df01eb46cfff5dbaa91d136ab47473f9105\": rpc error: code = NotFound desc = could not find container \"f37eeb0872bd53abc956a7e3019a5df01eb46cfff5dbaa91d136ab47473f9105\": container with ID starting with f37eeb0872bd53abc956a7e3019a5df01eb46cfff5dbaa91d136ab47473f9105 not found: ID does not exist"
Oct 30 22:30:03.637076 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:03.637026    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-779f32-29889990-m7sk5" podStartSLOduration=3.637012573 podStartE2EDuration="3.637012573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:03.636181571 +0000 UTC m=+2381.240489124" watchObservedRunningTime="2026-10-30 22:30:03.637012573 +0000 UTC m=+2381.241320126"
Oct 30 22:30:03.860691 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:03.860642    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bc714d-29889990-cxr7b" podStartSLOduration=3.860627483 podStartE2EDuration="3.860627483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:03.859245539 +0000 UTC m=+2381.463553092" watchObservedRunningTime="2026-10-30 22:30:03.860627483 +0000 UTC m=+2381.464935077"
Oct 30 22:30:03.969354 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:03.969317    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-jps22"]
Oct 30 22:30:03.975164 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:03.975135    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-jps22"]
Oct 30 22:30:04.329256 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:04.329228    5678 generic.go:356] "Generic (PLEG): container finished" podID="a94efb34-dd76-4be4-9872-947273ced7c0" containerID="02116764a900cfabdc3aa79cbaf8e1c7370de025ac47a3b8d8f72360a4d8c019" exitCode=60
Oct 30 22:30:04.329659 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:04.329317    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bc714d-29889990-cxr7b" event={"ID":"a94efb34-dd76-4be4-9872-947273ced7c0","Type":"ContainerDied","Data":"02116764a900cfabdc3aa79cbaf8e1c7370de025ac47a3b8d8f72360a4d8c019"}
Oct 30 22:30:04.329659 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:04.329395    5678 scope.go:122] "RemoveContainer" containerID="24f87e2877bac4409a1b79698dffc21362be2fafc4435931e4ea9fad31e155b6"
Oct 30 22:30:04.329783 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:04.329711    5678 scope.go:122] "RemoveContainer" containerID="02116764a900cfabdc3aa79cbaf8e1c7370de025ac47a3b8d8f72360a4d8c019"
Oct 30 22:30:04.330108 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:30:04.330078    5678 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-bc714d-29889990-cxr7b_odh-ai-gateway-infra(a94efb34-dd76-4be4-9872-947273ced7c0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bc714d-29889990-cxr7b" podUID="a94efb34-dd76-4be4-9872-947273ced7c0"
Oct 30 22:30:04.332855 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:04.332834    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c42c0a-29889990-z4dwc" event={"ID":"d0b1525a-699c-4121-94dd-f47d00b08ae7","Type":"ContainerStarted","Data":"4f8c1aeec04bdafebb85e02ed969995740383a101276d4fb37541d17ece390ff"}
Oct 30 22:30:04.335353 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:04.335328    5678 generic.go:356] "Generic (PLEG): container finished" podID="e8663b2d-a48a-49a1-a788-7de17d54e62b" containerID="95c0d0ca6b670b8c42ffaf41b1afcbdd5ecf9afbef96e392481596d9c473b274" exitCode=60
Oct 30 22:30:04.335589 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:04.335446    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb0b93-29889990-dldtk" event={"ID":"e8663b2d-a48a-49a1-a788-7de17d54e62b","Type":"ContainerDied","Data":"95c0d0ca6b670b8c42ffaf41b1afcbdd5ecf9afbef96e392481596d9c473b274"}
Oct 30 22:30:04.335708 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:04.335688    5678 scope.go:122] "RemoveContainer" containerID="95c0d0ca6b670b8c42ffaf41b1afcbdd5ecf9afbef96e392481596d9c473b274"
Oct 30 22:30:04.336000 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:30:04.335969    5678 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-fb0b93-29889990-dldtk_odh-ai-gateway-infra(e8663b2d-a48a-49a1-a788-7de17d54e62b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb0b93-29889990-dldtk" podUID="e8663b2d-a48a-49a1-a788-7de17d54e62b"
Oct 30 22:30:04.337610 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:04.337592    5678 scope.go:122] "RemoveContainer" containerID="bc9012044bb792f977418aa81796e9d46263075ea556fbd1e88c605b548f686d"
Oct 30 22:30:04.337846 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:30:04.337826    5678 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-99304b-29889990-wn2vq_odh-ai-gateway-infra(d886be7a-7208-446a-a3aa-6c478d277578)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-99304b-29889990-wn2vq" podUID="d886be7a-7208-446a-a3aa-6c478d277578"
Oct 30 22:30:04.339326 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:04.339304    5678 generic.go:356] "Generic (PLEG): container finished" podID="ddca5ef0-e1ab-493c-a3c8-a15e17d91d70" containerID="b33b64f81a78bc08d930e752afd8d204d7a9cc383139b21eb0864b5ac3a70922" exitCode=60
Oct 30 22:30:04.339563 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:04.339429    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-779f32-29889990-m7sk5" event={"ID":"ddca5ef0-e1ab-493c-a3c8-a15e17d91d70","Type":"ContainerDied","Data":"b33b64f81a78bc08d930e752afd8d204d7a9cc383139b21eb0864b5ac3a70922"}
Oct 30 22:30:04.339615 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:04.339562    5678 scope.go:122] "RemoveContainer" containerID="b33b64f81a78bc08d930e752afd8d204d7a9cc383139b21eb0864b5ac3a70922"
Oct 30 22:30:04.339858 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:30:04.339835    5678 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-779f32-29889990-m7sk5_odh-ai-gateway-infra(ddca5ef0-e1ab-493c-a3c8-a15e17d91d70)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-779f32-29889990-m7sk5" podUID="ddca5ef0-e1ab-493c-a3c8-a15e17d91d70"
Oct 30 22:30:04.403139 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:04.403117    5678 scope.go:122] "RemoveContainer" containerID="ac7337a3690e7b46dc3275ec0634385c8f1d59d68efff629d57ca134052a3551"
Oct 30 22:30:04.419639 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:04.419613    5678 scope.go:122] "RemoveContainer" containerID="aef0e96472e5e744dc0fd8090dc05ba59706057ab8809148f27827ccd43e4dd4"
Oct 30 22:30:04.528769 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:04.528733    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6373802-bc1e-4606-b946-4ae9753c560a" path="/var/lib/kubelet/pods/c6373802-bc1e-4606-b946-4ae9753c560a/volumes"
Oct 30 22:30:04.529341 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:04.529319    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d167328f-3684-41d7-9407-5ceec39dd03f" path="/var/lib/kubelet/pods/d167328f-3684-41d7-9407-5ceec39dd03f/volumes"
Oct 30 22:30:05.059321 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:05.059293    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-jsqpx"]
Oct 30 22:30:05.059611 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:05.059529    5678 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-jsqpx" podUID="bfbe138a-ae9f-45e4-9b30-1bf33a80929c" containerName="istio-proxy" containerID="cri-o://4e9990f35796c3fe9b7fd654106c892e4c66690f5b7599b08bf7fd91b07761aa" gracePeriod=30
Oct 30 22:30:05.063069 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:05.063041    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-x6j2w"]
Oct 30 22:30:05.063274 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:05.063245    5678 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-x6j2w" podUID="ea5fcd11-31d9-4836-92c3-00952b348844" containerName="istio-proxy" containerID="cri-o://dc67d0e32a6d45b2b0e6108b6394c96b0c5562fadd64fb273ad675b6f82d0152" gracePeriod=30
Oct 30 22:30:05.066634 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:05.066610    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xq4tp"]
Oct 30 22:30:05.066798 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:05.066779    5678 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xq4tp" podUID="b2fc93b9-e77d-4306-acec-85ea06f5c4be" containerName="istio-proxy" containerID="cri-o://e0b2c8c99a7b725e5c0ea317d2bd9569b40e2138e4ce49269cf5c7e79ba9b2a6" gracePeriod=30
Oct 30 22:30:05.305045 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:05.305013    5678 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-jsqpx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:30:05.305308 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:05.305064    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-jsqpx" podUID="bfbe138a-ae9f-45e4-9b30-1bf33a80929c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:30:05.348254 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:05.348177    5678 generic.go:356] "Generic (PLEG): container finished" podID="d0b1525a-699c-4121-94dd-f47d00b08ae7" containerID="4f8c1aeec04bdafebb85e02ed969995740383a101276d4fb37541d17ece390ff" exitCode=60
Oct 30 22:30:05.348683 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:05.348252    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c42c0a-29889990-z4dwc" event={"ID":"d0b1525a-699c-4121-94dd-f47d00b08ae7","Type":"ContainerDied","Data":"4f8c1aeec04bdafebb85e02ed969995740383a101276d4fb37541d17ece390ff"}
Oct 30 22:30:05.348683 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:05.348311    5678 scope.go:122] "RemoveContainer" containerID="8be3010819cd0b051c4469da99f7d6d6bf862c7452a5c8e974e71e880a2c7f06"
Oct 30 22:30:05.348683 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:05.348551    5678 scope.go:122] "RemoveContainer" containerID="4f8c1aeec04bdafebb85e02ed969995740383a101276d4fb37541d17ece390ff"
Oct 30 22:30:05.348861 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:30:05.348825    5678 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-c42c0a-29889990-z4dwc_odh-ai-gateway-infra(d0b1525a-699c-4121-94dd-f47d00b08ae7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c42c0a-29889990-z4dwc" podUID="d0b1525a-699c-4121-94dd-f47d00b08ae7"
Oct 30 22:30:05.350270 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:05.350252    5678 scope.go:122] "RemoveContainer" containerID="95c0d0ca6b670b8c42ffaf41b1afcbdd5ecf9afbef96e392481596d9c473b274"
Oct 30 22:30:05.350528 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:30:05.350509    5678 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-fb0b93-29889990-dldtk_odh-ai-gateway-infra(e8663b2d-a48a-49a1-a788-7de17d54e62b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb0b93-29889990-dldtk" podUID="e8663b2d-a48a-49a1-a788-7de17d54e62b"
Oct 30 22:30:05.351868 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:05.351848    5678 scope.go:122] "RemoveContainer" containerID="b33b64f81a78bc08d930e752afd8d204d7a9cc383139b21eb0864b5ac3a70922"
Oct 30 22:30:05.352084 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:30:05.352061    5678 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-779f32-29889990-m7sk5_odh-ai-gateway-infra(ddca5ef0-e1ab-493c-a3c8-a15e17d91d70)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-779f32-29889990-m7sk5" podUID="ddca5ef0-e1ab-493c-a3c8-a15e17d91d70"
Oct 30 22:30:05.353521 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:05.353504    5678 scope.go:122] "RemoveContainer" containerID="02116764a900cfabdc3aa79cbaf8e1c7370de025ac47a3b8d8f72360a4d8c019"
Oct 30 22:30:05.353718 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:30:05.353699    5678 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-bc714d-29889990-cxr7b_odh-ai-gateway-infra(a94efb34-dd76-4be4-9872-947273ced7c0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bc714d-29889990-cxr7b" podUID="a94efb34-dd76-4be4-9872-947273ced7c0"
Oct 30 22:30:06.362725 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:06.362693    5678 scope.go:122] "RemoveContainer" containerID="4f8c1aeec04bdafebb85e02ed969995740383a101276d4fb37541d17ece390ff"
Oct 30 22:30:06.363112 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:30:06.363054    5678 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-c42c0a-29889990-z4dwc_odh-ai-gateway-infra(d0b1525a-699c-4121-94dd-f47d00b08ae7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c42c0a-29889990-z4dwc" podUID="d0b1525a-699c-4121-94dd-f47d00b08ae7"
Oct 30 22:30:08.485951 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:08.485911    5678 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-x6j2w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:30:08.486395 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:08.485974    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-x6j2w" podUID="ea5fcd11-31d9-4836-92c3-00952b348844" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:30:08.747042 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:08.746969    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-75fb62-845c55c77b-gmd6f"]
Oct 30 22:30:08.747671 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:08.747653    5678 cpu_manager.go:414] "containerMap: removing container" podUID="c6373802-bc1e-4606-b946-4ae9753c560a" containerName="istio-proxy"
Oct 30 22:30:08.747744 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:08.747671    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6373802-bc1e-4606-b946-4ae9753c560a" containerName="istio-proxy"
Oct 30 22:30:08.747744 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:08.747715    5678 cpu_manager.go:414] "containerMap: removing container" podUID="d167328f-3684-41d7-9407-5ceec39dd03f" containerName="istio-proxy"
Oct 30 22:30:08.747744 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:08.747721    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d167328f-3684-41d7-9407-5ceec39dd03f" containerName="istio-proxy"
Oct 30 22:30:08.747873 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:08.747817    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="d167328f-3684-41d7-9407-5ceec39dd03f" containerName="istio-proxy"
Oct 30 22:30:08.747873 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:08.747830    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6373802-bc1e-4606-b946-4ae9753c560a" containerName="istio-proxy"
Oct 30 22:30:08.752702 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:08.752681    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-75fb62-845c55c77b-gmd6f"
Oct 30 22:30:08.790808 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:08.790779    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwb64\" (UniqueName: \"kubernetes.io/projected/0e40fbc5-668a-4917-9344-dea69c625d26-kube-api-access-bwb64\") pod \"maas-api-e2e-inf-a-75fb62-845c55c77b-gmd6f\" (UID: \"0e40fbc5-668a-4917-9344-dea69c625d26\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-75fb62-845c55c77b-gmd6f"
Oct 30 22:30:08.790938 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:08.790836    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e40fbc5-668a-4917-9344-dea69c625d26-openshift-service-ca\") pod \"maas-api-e2e-inf-a-75fb62-845c55c77b-gmd6f\" (UID: \"0e40fbc5-668a-4917-9344-dea69c625d26\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-75fb62-845c55c77b-gmd6f"
Oct 30 22:30:08.790938 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:08.790903    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e40fbc5-668a-4917-9344-dea69c625d26-maas-api-tls\") pod \"maas-api-e2e-inf-a-75fb62-845c55c77b-gmd6f\" (UID: \"0e40fbc5-668a-4917-9344-dea69c625d26\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-75fb62-845c55c77b-gmd6f"
Oct 30 22:30:08.791037 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:08.791015    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e40fbc5-668a-4917-9344-dea69c625d26-metrics-certs\") pod \"maas-api-e2e-inf-a-75fb62-845c55c77b-gmd6f\" (UID: \"0e40fbc5-668a-4917-9344-dea69c625d26\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-75fb62-845c55c77b-gmd6f"
Oct 30 22:30:08.851392 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:08.851365    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-75fb62-serving-cert\""
Oct 30 22:30:08.855071 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:08.855041    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-75fb62-845c55c77b-gmd6f"]
Oct 30 22:30:08.891993 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:08.891958    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwb64\" (UniqueName: \"kubernetes.io/projected/0e40fbc5-668a-4917-9344-dea69c625d26-kube-api-access-bwb64\") pod \"maas-api-e2e-inf-a-75fb62-845c55c77b-gmd6f\" (UID: \"0e40fbc5-668a-4917-9344-dea69c625d26\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-75fb62-845c55c77b-gmd6f"
Oct 30 22:30:08.892185 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:08.892054    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e40fbc5-668a-4917-9344-dea69c625d26-openshift-service-ca\") pod \"maas-api-e2e-inf-a-75fb62-845c55c77b-gmd6f\" (UID: \"0e40fbc5-668a-4917-9344-dea69c625d26\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-75fb62-845c55c77b-gmd6f"
Oct 30 22:30:08.892185 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:08.892135    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e40fbc5-668a-4917-9344-dea69c625d26-maas-api-tls\") pod \"maas-api-e2e-inf-a-75fb62-845c55c77b-gmd6f\" (UID: \"0e40fbc5-668a-4917-9344-dea69c625d26\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-75fb62-845c55c77b-gmd6f"
Oct 30 22:30:08.892314 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:08.892243    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e40fbc5-668a-4917-9344-dea69c625d26-metrics-certs\") pod \"maas-api-e2e-inf-a-75fb62-845c55c77b-gmd6f\" (UID: \"0e40fbc5-668a-4917-9344-dea69c625d26\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-75fb62-845c55c77b-gmd6f"
Oct 30 22:30:08.892567 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:30:08.892545    5678 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-75fb62-serving-cert: secret "maas-api-e2e-inf-a-75fb62-serving-cert" not found
Oct 30 22:30:08.892763 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:30:08.892742    5678 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e40fbc5-668a-4917-9344-dea69c625d26-maas-api-tls podName:0e40fbc5-668a-4917-9344-dea69c625d26 nodeName:}" failed. No retries permitted until 2026-10-30 22:30:09.39272185 +0000 UTC m=+2386.997029399 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0e40fbc5-668a-4917-9344-dea69c625d26-maas-api-tls") pod "maas-api-e2e-inf-a-75fb62-845c55c77b-gmd6f" (UID: "0e40fbc5-668a-4917-9344-dea69c625d26") : secret "maas-api-e2e-inf-a-75fb62-serving-cert" not found
Oct 30 22:30:08.893053 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:08.893028    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e40fbc5-668a-4917-9344-dea69c625d26-openshift-service-ca\") pod \"maas-api-e2e-inf-a-75fb62-845c55c77b-gmd6f\" (UID: \"0e40fbc5-668a-4917-9344-dea69c625d26\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-75fb62-845c55c77b-gmd6f"
Oct 30 22:30:08.895156 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:08.895124    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e40fbc5-668a-4917-9344-dea69c625d26-metrics-certs\") pod \"maas-api-e2e-inf-a-75fb62-845c55c77b-gmd6f\" (UID: \"0e40fbc5-668a-4917-9344-dea69c625d26\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-75fb62-845c55c77b-gmd6f"
Oct 30 22:30:08.951018 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:08.950996    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwb64\" (UniqueName: \"kubernetes.io/projected/0e40fbc5-668a-4917-9344-dea69c625d26-kube-api-access-bwb64\") pod \"maas-api-e2e-inf-a-75fb62-845c55c77b-gmd6f\" (UID: \"0e40fbc5-668a-4917-9344-dea69c625d26\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-75fb62-845c55c77b-gmd6f"
Oct 30 22:30:09.400225 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:09.400187    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e40fbc5-668a-4917-9344-dea69c625d26-maas-api-tls\") pod \"maas-api-e2e-inf-a-75fb62-845c55c77b-gmd6f\" (UID: \"0e40fbc5-668a-4917-9344-dea69c625d26\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-75fb62-845c55c77b-gmd6f"
Oct 30 22:30:09.402779 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:09.402753    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e40fbc5-668a-4917-9344-dea69c625d26-maas-api-tls\") pod \"maas-api-e2e-inf-a-75fb62-845c55c77b-gmd6f\" (UID: \"0e40fbc5-668a-4917-9344-dea69c625d26\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-75fb62-845c55c77b-gmd6f"
Oct 30 22:30:09.666408 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:09.666314    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-75fb62-845c55c77b-gmd6f"
Oct 30 22:30:09.770485 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:09.770445    5678 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-xq4tp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:30:09.770659 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:09.770515    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xq4tp" podUID="b2fc93b9-e77d-4306-acec-85ea06f5c4be" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:30:10.063819 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:10.063740    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-75fb62-845c55c77b-gmd6f"]
Oct 30 22:30:10.070715 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:30:10.070680    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e40fbc5_668a_4917_9344_dea69c625d26.slice/crio-fbb4a69e0edfb3965283f031a5bc887d08bf5e3cd9c80912801b985ad6ced1f7 WatchSource:0}: Error finding container fbb4a69e0edfb3965283f031a5bc887d08bf5e3cd9c80912801b985ad6ced1f7: Status 404 returned error can't find the container with id fbb4a69e0edfb3965283f031a5bc887d08bf5e3cd9c80912801b985ad6ced1f7
Oct 30 22:30:10.072368 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:10.072348    5678 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 22:30:10.352689 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:10.352642    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-jsqpx"
Oct 30 22:30:10.376767 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:10.376743    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-x6j2w"
Oct 30 22:30:10.396762 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:10.396730    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-75fb62-845c55c77b-gmd6f" event={"ID":"0e40fbc5-668a-4917-9344-dea69c625d26","Type":"ContainerStarted","Data":"fbb4a69e0edfb3965283f031a5bc887d08bf5e3cd9c80912801b985ad6ced1f7"}
Oct 30 22:30:10.398279 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:10.398252    5678 generic.go:356] "Generic (PLEG): container finished" podID="bfbe138a-ae9f-45e4-9b30-1bf33a80929c" containerID="4e9990f35796c3fe9b7fd654106c892e4c66690f5b7599b08bf7fd91b07761aa" exitCode=0
Oct 30 22:30:10.398378 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:10.398295    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-jsqpx"
Oct 30 22:30:10.398378 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:10.398333    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-jsqpx" event={"ID":"bfbe138a-ae9f-45e4-9b30-1bf33a80929c","Type":"ContainerDied","Data":"4e9990f35796c3fe9b7fd654106c892e4c66690f5b7599b08bf7fd91b07761aa"}
Oct 30 22:30:10.398518 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:10.398385    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-jsqpx" event={"ID":"bfbe138a-ae9f-45e4-9b30-1bf33a80929c","Type":"ContainerDied","Data":"d54d4a22287506a666214b9dad819e79e31e3cfd6ed010430604262cc8d01854"}
Oct 30 22:30:10.398518 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:10.398453    5678 scope.go:122] "RemoveContainer" containerID="4e9990f35796c3fe9b7fd654106c892e4c66690f5b7599b08bf7fd91b07761aa"
Oct 30 22:30:10.399862 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:10.399840    5678 generic.go:356] "Generic (PLEG): container finished" podID="ea5fcd11-31d9-4836-92c3-00952b348844" containerID="dc67d0e32a6d45b2b0e6108b6394c96b0c5562fadd64fb273ad675b6f82d0152" exitCode=0
Oct 30 22:30:10.399982 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:10.399882    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-x6j2w"
Oct 30 22:30:10.399982 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:10.399885    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-x6j2w" event={"ID":"ea5fcd11-31d9-4836-92c3-00952b348844","Type":"ContainerDied","Data":"dc67d0e32a6d45b2b0e6108b6394c96b0c5562fadd64fb273ad675b6f82d0152"}
Oct 30 22:30:10.399982 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:10.399929    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-x6j2w" event={"ID":"ea5fcd11-31d9-4836-92c3-00952b348844","Type":"ContainerDied","Data":"5cc3b2e3b5c564418e6878cf2c41363c99529405726380861281c898438a1af1"}
Oct 30 22:30:10.401823 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:10.401801    5678 generic.go:356] "Generic (PLEG): container finished" podID="b2fc93b9-e77d-4306-acec-85ea06f5c4be" containerID="e0b2c8c99a7b725e5c0ea317d2bd9569b40e2138e4ce49269cf5c7e79ba9b2a6" exitCode=0
Oct 30 22:30:10.401914 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:10.401882    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xq4tp" event={"ID":"b2fc93b9-e77d-4306-acec-85ea06f5c4be","Type":"ContainerDied","Data":"e0b2c8c99a7b725e5c0ea317d2bd9569b40e2138e4ce49269cf5c7e79ba9b2a6"}
Oct 30 22:30:10.409040 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:10.409021    5678 scope.go:122] "RemoveContainer" containerID="4e9990f35796c3fe9b7fd654106c892e4c66690f5b7599b08bf7fd91b07761aa"
Oct 30 22:30:10.409351 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:30:10.409332    5678 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e9990f35796c3fe9b7fd654106c892e4c66690f5b7599b08bf7fd91b07761aa\": container with ID starting with 4e9990f35796c3fe9b7fd654106c892e4c66690f5b7599b08bf7fd91b07761aa not found: ID does not exist" containerID="4e9990f35796c3fe9b7fd654106c892e4c66690f5b7599b08bf7fd91b07761aa"
Oct 30 22:30:10.409412 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:10.409358    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e9990f35796c3fe9b7fd654106c892e4c66690f5b7599b08bf7fd91b07761aa"} err="failed to get container status \"4e9990f35796c3fe9b7fd654106c892e4c66690f5b7599b08bf7fd91b07761aa\": rpc error: code = NotFound desc = could not find container \"4e9990f35796c3fe9b7fd654106c892e4c66690f5b7599b08bf7fd91b07761aa\": container with ID starting with 4e9990f35796c3fe9b7fd654106c892e4c66690f5b7599b08bf7fd91b07761aa not found: ID does not exist"
Oct 30 22:30:10.409412 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:10.409377    5678 scope.go:122] "RemoveContainer" containerID="dc67d0e32a6d45b2b0e6108b6394c96b0c5562fadd64fb273ad675b6f82d0152"
Oct 30 22:30:10.412052 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:10.412027    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfbe138a-ae9f-45e4-9b30-1bf33a80929c-kube-api-access-8q4gn\" (UniqueName: \"kubernetes.io/projected/bfbe138a-ae9f-45e4-9b30-1bf33a80929c-kube-api-access-8q4gn\") pod \"bfbe138a-ae9f-45e4-9b30-1bf33a80929c\" (UID: \"bfbe138a-ae9f-45e4-9b30-1bf33a80929c\") "
Oct 30 22:30:10.412159 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:10.412107    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfbe138a-ae9f-45e4-9b30-1bf33a80929c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfbe138a-ae9f-45e4-9b30-1bf33a80929c-credential-socket\") pod \"bfbe138a-ae9f-45e4-9b30-1bf33a80929c\" (UID: \"bfbe138a-ae9f-45e4-9b30-1bf33a80929c\") "
Oct 30 22:30:10.412159 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:10.412142    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bfbe138a-ae9f-45e4-9b30-1bf33a80929c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfbe138a-ae9f-45e4-9b30-1bf33a80929c-istio-podinfo\") pod \"bfbe138a-ae9f-45e4-9b30-1bf33a80929c\" (UID: \"bfbe138a-ae9f-45e4-9b30-1bf33a80929c\") "
Oct 30 22:30:10.412276 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:10.412221    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfbe138a-ae9f-45e4-9b30-1bf33a80929c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfbe138a-ae9f-45e4-9b30-1bf33a80929c-istiod-ca-cert\") pod \"bfbe138a-ae9f-45e4-9b30-1bf33a80929c\" (UID: \"bfbe138a-ae9f-45e4-9b30-1bf33a80929c\") "
Oct 30 22:30:10.412276 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:10.412258    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfbe138a-ae9f-45e4-9b30-1bf33a80929c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfbe138a-ae9f-45e4-9b30-1bf33a80929c-istio-envoy\") pod \"bfbe138a-ae9f-45e4-9b30-1bf33a80929c\" (UID: \"bfbe138a-ae9f-45e4-9b30-1bf33a80929c\") "
Oct 30 22:30:10.412382 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:10.412311    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfbe138a-ae9f-45e4-9b30-1bf33a80929c-istio-token\" (UniqueName: \"kubernetes.io/projected/bfbe138a-ae9f-45e4-9b30-1bf33a80929c-istio-token\") pod \"bfbe138a-ae9f-45e4-9b30-1bf33a80929c\" (UID: \"bfbe138a-ae9f-45e4-9b30-1bf33a80929c\") "
Oct 30 22:30:10.412454 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:10.412374    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfbe138a-ae9f-45e4-9b30-1bf33a80929c-credential-socket" pod "bfbe138a-ae9f-45e4-9b30-1bf33a80929c" (UID: "bfbe138a-ae9f-45e4-9b30-1bf33a80929c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:10.412520 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:10.412482    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfbe138a-ae9f-45e4-9b30-1bf33a80929c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfbe138a-ae9f-45e4-9b30-1bf33a80929c-workload-socket\") pod \"bfbe138a-ae9f-45e4-9b30-1bf33a80929c\" (UID: \"bfbe138a-ae9f-45e4-9b30-1bf33a80929c\") "
Oct 30 22:30:10.412578 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:10.412550    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfbe138a-ae9f-45e4-9b30-1bf33a80929c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfbe138a-ae9f-45e4-9b30-1bf33a80929c-istio-data\") pod \"bfbe138a-ae9f-45e4-9b30-1bf33a80929c\" (UID: \"bfbe138a-ae9f-45e4-9b30-1bf33a80929c\") "
Oct 30 22:30:10.412700 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:10.412675    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfbe138a-ae9f-45e4-9b30-1bf33a80929c-istiod-ca-cert" pod "bfbe138a-ae9f-45e4-9b30-1bf33a80929c" (UID: "bfbe138a-ae9f-45e4-9b30-1bf33a80929c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:10.412852 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:10.412832    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfbe138a-ae9f-45e4-9b30-1bf33a80929c-workload-socket" pod "bfbe138a-ae9f-45e4-9b30-1bf33a80929c" (UID: "bfbe138a-ae9f-45e4-9b30-1bf33a80929c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:10.413212 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:10.413126    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfbe138a-ae9f-45e4-9b30-1bf33a80929c-istio-data" pod "bfbe138a-ae9f-45e4-9b30-1bf33a80929c" (UID: "bfbe138a-ae9f-45e4-9b30-1bf33a80929c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:10.413630 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:10.413294    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfbe138a-ae9f-45e4-9b30-1bf33a80929c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfbe138a-ae9f-45e4-9b30-1bf33a80929c-workload-certs\") pod \"bfbe138a-ae9f-45e4-9b30-1bf33a80929c\" (UID: \"bfbe138a-ae9f-45e4-9b30-1bf33a80929c\") "
Oct 30 22:30:10.413630 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:10.413619    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfbe138a-ae9f-45e4-9b30-1bf33a80929c-workload-certs" pod "bfbe138a-ae9f-45e4-9b30-1bf33a80929c" (UID: "bfbe138a-ae9f-45e4-9b30-1bf33a80929c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:10.414285 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:10.414263    5678 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfbe138a-ae9f-45e4-9b30-1bf33a80929c-workload-certs\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:10.414285 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:10.414284    5678 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfbe138a-ae9f-45e4-9b30-1bf33a80929c-credential-socket\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:10.414456 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:10.414295    5678 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfbe138a-ae9f-45e4-9b30-1bf33a80929c-istiod-ca-cert\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:10.414456 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:10.414303    5678 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfbe138a-ae9f-45e4-9b30-1bf33a80929c-workload-socket\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:10.414456 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:10.414312    5678 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfbe138a-ae9f-45e4-9b30-1bf33a80929c-istio-data\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:10.414757 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:10.414731    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bfbe138a-ae9f-45e4-9b30-1bf33a80929c-istio-podinfo" pod "bfbe138a-ae9f-45e4-9b30-1bf33a80929c" (UID: "bfbe138a-ae9f-45e4-9b30-1bf33a80929c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:30:10.414863 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:10.414779    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfbe138a-ae9f-45e4-9b30-1bf33a80929c-kube-api-access-8q4gn" pod "bfbe138a-ae9f-45e4-9b30-1bf33a80929c" (UID: "bfbe138a-ae9f-45e4-9b30-1bf33a80929c"). InnerVolumeSpecName "kube-api-access-8q4gn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:10.414973 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:10.414954    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfbe138a-ae9f-45e4-9b30-1bf33a80929c-istio-token" pod "bfbe138a-ae9f-45e4-9b30-1bf33a80929c" (UID: "bfbe138a-ae9f-45e4-9b30-1bf33a80929c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:10.415753 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:10.415731    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfbe138a-ae9f-45e4-9b30-1bf33a80929c-istio-envoy" pod "bfbe138a-ae9f-45e4-9b30-1bf33a80929c" (UID: "bfbe138a-ae9f-45e4-9b30-1bf33a80929c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:10.419812 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:10.419794    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xq4tp"
Oct 30 22:30:10.421378 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:10.421362    5678 scope.go:122] "RemoveContainer" containerID="dc67d0e32a6d45b2b0e6108b6394c96b0c5562fadd64fb273ad675b6f82d0152"
Oct 30 22:30:10.421690 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:30:10.421671    5678 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc67d0e32a6d45b2b0e6108b6394c96b0c5562fadd64fb273ad675b6f82d0152\": container with ID starting with dc67d0e32a6d45b2b0e6108b6394c96b0c5562fadd64fb273ad675b6f82d0152 not found: ID does not exist" containerID="dc67d0e32a6d45b2b0e6108b6394c96b0c5562fadd64fb273ad675b6f82d0152"
Oct 30 22:30:10.421803 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:10.421697    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc67d0e32a6d45b2b0e6108b6394c96b0c5562fadd64fb273ad675b6f82d0152"} err="failed to get container status \"dc67d0e32a6d45b2b0e6108b6394c96b0c5562fadd64fb273ad675b6f82d0152\": rpc error: code = NotFound desc = could not find container \"dc67d0e32a6d45b2b0e6108b6394c96b0c5562fadd64fb273ad675b6f82d0152\": container with ID starting with dc67d0e32a6d45b2b0e6108b6394c96b0c5562fadd64fb273ad675b6f82d0152 not found: ID does not exist"
Oct 30 22:30:10.515012 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:10.514983    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2fc93b9-e77d-4306-acec-85ea06f5c4be-kube-api-access-mjrwz\" (UniqueName: \"kubernetes.io/projected/b2fc93b9-e77d-4306-acec-85ea06f5c4be-kube-api-access-mjrwz\") pod \"b2fc93b9-e77d-4306-acec-85ea06f5c4be\" (UID: \"b2fc93b9-e77d-4306-acec-85ea06f5c4be\") "
Oct 30 22:30:10.515164 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:10.515088    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2fc93b9-e77d-4306-acec-85ea06f5c4be-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2fc93b9-e77d-4306-acec-85ea06f5c4be-istio-data\") pod \"b2fc93b9-e77d-4306-acec-85ea06f5c4be\" (UID: \"b2fc93b9-e77d-4306-acec-85ea06f5c4be\") "
Oct 30 22:30:10.515164 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:10.515113    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea5fcd11-31d9-4836-92c3-00952b348844-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea5fcd11-31d9-4836-92c3-00952b348844-istio-envoy\") pod \"ea5fcd11-31d9-4836-92c3-00952b348844\" (UID: \"ea5fcd11-31d9-4836-92c3-00952b348844\") "
Oct 30 22:30:10.515164 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:10.515139    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea5fcd11-31d9-4836-92c3-00952b348844-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea5fcd11-31d9-4836-92c3-00952b348844-credential-socket\") pod \"ea5fcd11-31d9-4836-92c3-00952b348844\" (UID: \"ea5fcd11-31d9-4836-92c3-00952b348844\") "
Oct 30 22:30:10.515348 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:10.515197    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea5fcd11-31d9-4836-92c3-00952b348844-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea5fcd11-31d9-4836-92c3-00952b348844-workload-certs\") pod \"ea5fcd11-31d9-4836-92c3-00952b348844\" (UID: \"ea5fcd11-31d9-4836-92c3-00952b348844\") "
Oct 30 22:30:10.515348 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:10.515263    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ea5fcd11-31d9-4836-92c3-00952b348844-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea5fcd11-31d9-4836-92c3-00952b348844-istio-podinfo\") pod \"ea5fcd11-31d9-4836-92c3-00952b348844\" (UID: \"ea5fcd11-31d9-4836-92c3-00952b348844\") "
Oct 30 22:30:10.515348 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:10.515308    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2fc93b9-e77d-4306-acec-85ea06f5c4be-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2fc93b9-e77d-4306-acec-85ea06f5c4be-workload-certs\") pod \"b2fc93b9-e77d-4306-acec-85ea06f5c4be\" (UID: \"b2fc93b9-e77d-4306-acec-85ea06f5c4be\") "
Oct 30 22:30:10.515566 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:10.515348    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2fc93b9-e77d-4306-acec-85ea06f5c4be-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2fc93b9-e77d-4306-acec-85ea06f5c4be-istiod-ca-cert\") pod \"b2fc93b9-e77d-4306-acec-85ea06f5c4be\" (UID: \"b2fc93b9-e77d-4306-acec-85ea06f5c4be\") "
Oct 30 22:30:10.515566 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:10.515443    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea5fcd11-31d9-4836-92c3-00952b348844-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea5fcd11-31d9-4836-92c3-00952b348844-istiod-ca-cert\") pod \"ea5fcd11-31d9-4836-92c3-00952b348844\" (UID: \"ea5fcd11-31d9-4836-92c3-00952b348844\") "
Oct 30 22:30:10.515566 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:10.515466    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2fc93b9-e77d-4306-acec-85ea06f5c4be-istio-data" pod "b2fc93b9-e77d-4306-acec-85ea06f5c4be" (UID: "b2fc93b9-e77d-4306-acec-85ea06f5c4be"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:10.515566 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:10.515503    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea5fcd11-31d9-4836-92c3-00952b348844-istio-token\" (UniqueName: \"kubernetes.io/projected/ea5fcd11-31d9-4836-92c3-00952b348844-istio-token\") pod \"ea5fcd11-31d9-4836-92c3-00952b348844\" (UID: \"ea5fcd11-31d9-4836-92c3-00952b348844\") "
Oct 30 22:30:10.515566 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:10.515544    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea5fcd11-31d9-4836-92c3-00952b348844-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea5fcd11-31d9-4836-92c3-00952b348844-istio-data\") pod \"ea5fcd11-31d9-4836-92c3-00952b348844\" (UID: \"ea5fcd11-31d9-4836-92c3-00952b348844\") "
Oct 30 22:30:10.515812 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:10.515581    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2fc93b9-e77d-4306-acec-85ea06f5c4be-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2fc93b9-e77d-4306-acec-85ea06f5c4be-workload-socket\") pod \"b2fc93b9-e77d-4306-acec-85ea06f5c4be\" (UID: \"b2fc93b9-e77d-4306-acec-85ea06f5c4be\") "
Oct 30 22:30:10.515812 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:10.515614    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b2fc93b9-e77d-4306-acec-85ea06f5c4be-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2fc93b9-e77d-4306-acec-85ea06f5c4be-istio-podinfo\") pod \"b2fc93b9-e77d-4306-acec-85ea06f5c4be\" (UID: \"b2fc93b9-e77d-4306-acec-85ea06f5c4be\") "
Oct 30 22:30:10.515812 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:10.515652    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea5fcd11-31d9-4836-92c3-00952b348844-kube-api-access-mp69h\" (UniqueName: \"kubernetes.io/projected/ea5fcd11-31d9-4836-92c3-00952b348844-kube-api-access-mp69h\") pod \"ea5fcd11-31d9-4836-92c3-00952b348844\" (UID: \"ea5fcd11-31d9-4836-92c3-00952b348844\") "
Oct 30 22:30:10.515812 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:10.515633    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea5fcd11-31d9-4836-92c3-00952b348844-credential-socket" pod "ea5fcd11-31d9-4836-92c3-00952b348844" (UID: "ea5fcd11-31d9-4836-92c3-00952b348844"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:10.515812 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:10.515708    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea5fcd11-31d9-4836-92c3-00952b348844-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea5fcd11-31d9-4836-92c3-00952b348844-workload-socket\") pod \"ea5fcd11-31d9-4836-92c3-00952b348844\" (UID: \"ea5fcd11-31d9-4836-92c3-00952b348844\") "
Oct 30 22:30:10.515812 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:10.515710    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2fc93b9-e77d-4306-acec-85ea06f5c4be-workload-certs" pod "b2fc93b9-e77d-4306-acec-85ea06f5c4be" (UID: "b2fc93b9-e77d-4306-acec-85ea06f5c4be"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:10.515812 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:10.515738    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2fc93b9-e77d-4306-acec-85ea06f5c4be-istio-token\" (UniqueName: \"kubernetes.io/projected/b2fc93b9-e77d-4306-acec-85ea06f5c4be-istio-token\") pod \"b2fc93b9-e77d-4306-acec-85ea06f5c4be\" (UID: \"b2fc93b9-e77d-4306-acec-85ea06f5c4be\") "
Oct 30 22:30:10.516232 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:10.515812    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2fc93b9-e77d-4306-acec-85ea06f5c4be-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2fc93b9-e77d-4306-acec-85ea06f5c4be-istio-envoy\") pod \"b2fc93b9-e77d-4306-acec-85ea06f5c4be\" (UID: \"b2fc93b9-e77d-4306-acec-85ea06f5c4be\") "
Oct 30 22:30:10.516232 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:10.515853    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2fc93b9-e77d-4306-acec-85ea06f5c4be-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2fc93b9-e77d-4306-acec-85ea06f5c4be-credential-socket\") pod \"b2fc93b9-e77d-4306-acec-85ea06f5c4be\" (UID: \"b2fc93b9-e77d-4306-acec-85ea06f5c4be\") "
Oct 30 22:30:10.516232 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:10.516009    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2fc93b9-e77d-4306-acec-85ea06f5c4be-istiod-ca-cert" pod "b2fc93b9-e77d-4306-acec-85ea06f5c4be" (UID: "b2fc93b9-e77d-4306-acec-85ea06f5c4be"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:10.516391 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:10.516235    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea5fcd11-31d9-4836-92c3-00952b348844-workload-socket" pod "ea5fcd11-31d9-4836-92c3-00952b348844" (UID: "ea5fcd11-31d9-4836-92c3-00952b348844"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:10.516391 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:10.516276    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea5fcd11-31d9-4836-92c3-00952b348844-istiod-ca-cert" pod "ea5fcd11-31d9-4836-92c3-00952b348844" (UID: "ea5fcd11-31d9-4836-92c3-00952b348844"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:10.516522 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:10.516434    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea5fcd11-31d9-4836-92c3-00952b348844-istio-data" pod "ea5fcd11-31d9-4836-92c3-00952b348844" (UID: "ea5fcd11-31d9-4836-92c3-00952b348844"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:10.516620 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:10.516601    5678 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfbe138a-ae9f-45e4-9b30-1bf33a80929c-istio-podinfo\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:10.516689 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:10.516623    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2fc93b9-e77d-4306-acec-85ea06f5c4be-workload-socket" pod "b2fc93b9-e77d-4306-acec-85ea06f5c4be" (UID: "b2fc93b9-e77d-4306-acec-85ea06f5c4be"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:10.516689 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:10.516626    5678 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2fc93b9-e77d-4306-acec-85ea06f5c4be-istio-data\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:10.516689 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:10.516661    5678 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea5fcd11-31d9-4836-92c3-00952b348844-credential-socket\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:10.516689 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:10.516679    5678 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfbe138a-ae9f-45e4-9b30-1bf33a80929c-istio-envoy\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:10.516918 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:10.516693    5678 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfbe138a-ae9f-45e4-9b30-1bf33a80929c-istio-token\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:10.516918 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:10.516706    5678 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2fc93b9-e77d-4306-acec-85ea06f5c4be-workload-certs\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:10.516918 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:10.516719    5678 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2fc93b9-e77d-4306-acec-85ea06f5c4be-istiod-ca-cert\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:10.516918 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:10.516733    5678 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea5fcd11-31d9-4836-92c3-00952b348844-istiod-ca-cert\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:10.516918 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:10.516747    5678 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea5fcd11-31d9-4836-92c3-00952b348844-istio-data\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:10.516918 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:10.516763    5678 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea5fcd11-31d9-4836-92c3-00952b348844-workload-socket\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:10.516918 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:10.516778    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8q4gn\" (UniqueName: \"kubernetes.io/projected/bfbe138a-ae9f-45e4-9b30-1bf33a80929c-kube-api-access-8q4gn\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:10.520030 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:10.519662    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ea5fcd11-31d9-4836-92c3-00952b348844-istio-podinfo" pod "ea5fcd11-31d9-4836-92c3-00952b348844" (UID: "ea5fcd11-31d9-4836-92c3-00952b348844"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:30:10.520030 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:10.519735    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b2fc93b9-e77d-4306-acec-85ea06f5c4be-istio-podinfo" pod "b2fc93b9-e77d-4306-acec-85ea06f5c4be" (UID: "b2fc93b9-e77d-4306-acec-85ea06f5c4be"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:30:10.520030 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:10.519744    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2fc93b9-e77d-4306-acec-85ea06f5c4be-istio-token" pod "b2fc93b9-e77d-4306-acec-85ea06f5c4be" (UID: "b2fc93b9-e77d-4306-acec-85ea06f5c4be"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:10.520030 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:10.519782    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea5fcd11-31d9-4836-92c3-00952b348844-istio-token" pod "ea5fcd11-31d9-4836-92c3-00952b348844" (UID: "ea5fcd11-31d9-4836-92c3-00952b348844"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:10.520030 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:10.519893    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea5fcd11-31d9-4836-92c3-00952b348844-istio-envoy" pod "ea5fcd11-31d9-4836-92c3-00952b348844" (UID: "ea5fcd11-31d9-4836-92c3-00952b348844"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:10.520030 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:10.520004    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea5fcd11-31d9-4836-92c3-00952b348844-workload-certs" pod "ea5fcd11-31d9-4836-92c3-00952b348844" (UID: "ea5fcd11-31d9-4836-92c3-00952b348844"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:10.520030 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:10.520023    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2fc93b9-e77d-4306-acec-85ea06f5c4be-istio-envoy" pod "b2fc93b9-e77d-4306-acec-85ea06f5c4be" (UID: "b2fc93b9-e77d-4306-acec-85ea06f5c4be"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:10.520526 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:10.520504    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2fc93b9-e77d-4306-acec-85ea06f5c4be-kube-api-access-mjrwz" pod "b2fc93b9-e77d-4306-acec-85ea06f5c4be" (UID: "b2fc93b9-e77d-4306-acec-85ea06f5c4be"). InnerVolumeSpecName "kube-api-access-mjrwz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:10.520625 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:10.520585    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2fc93b9-e77d-4306-acec-85ea06f5c4be-credential-socket" pod "b2fc93b9-e77d-4306-acec-85ea06f5c4be" (UID: "b2fc93b9-e77d-4306-acec-85ea06f5c4be"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:10.521973 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:10.521954    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea5fcd11-31d9-4836-92c3-00952b348844-kube-api-access-mp69h" pod "ea5fcd11-31d9-4836-92c3-00952b348844" (UID: "ea5fcd11-31d9-4836-92c3-00952b348844"). InnerVolumeSpecName "kube-api-access-mp69h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:10.574846 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:30:10.574771    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 22:30:10.618480 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:10.618441    5678 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2fc93b9-e77d-4306-acec-85ea06f5c4be-workload-socket\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:10.618480 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:10.618481    5678 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2fc93b9-e77d-4306-acec-85ea06f5c4be-istio-podinfo\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:10.618789 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:10.618499    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mp69h\" (UniqueName: \"kubernetes.io/projected/ea5fcd11-31d9-4836-92c3-00952b348844-kube-api-access-mp69h\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:10.618789 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:10.618515    5678 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2fc93b9-e77d-4306-acec-85ea06f5c4be-istio-token\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:10.618789 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:10.618529    5678 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2fc93b9-e77d-4306-acec-85ea06f5c4be-istio-envoy\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:10.618789 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:10.618543    5678 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2fc93b9-e77d-4306-acec-85ea06f5c4be-credential-socket\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:10.618789 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:10.618561    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjrwz\" (UniqueName: \"kubernetes.io/projected/b2fc93b9-e77d-4306-acec-85ea06f5c4be-kube-api-access-mjrwz\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:10.618789 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:10.618576    5678 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea5fcd11-31d9-4836-92c3-00952b348844-istio-envoy\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:10.618789 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:10.618593    5678 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea5fcd11-31d9-4836-92c3-00952b348844-workload-certs\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:10.618789 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:10.618606    5678 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea5fcd11-31d9-4836-92c3-00952b348844-istio-podinfo\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:10.618789 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:10.618619    5678 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea5fcd11-31d9-4836-92c3-00952b348844-istio-token\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:10.864667 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:10.864642    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-jsqpx"]
Oct 30 22:30:10.885285 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:10.885251    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-jsqpx"]
Oct 30 22:30:10.988797 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:10.988726    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-x6j2w"]
Oct 30 22:30:10.993113 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:10.993086    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-x6j2w"]
Oct 30 22:30:11.411577 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:11.411544    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xq4tp"
Oct 30 22:30:11.411761 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:11.411594    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xq4tp" event={"ID":"b2fc93b9-e77d-4306-acec-85ea06f5c4be","Type":"ContainerDied","Data":"5e55ed36e69b6dfb8ce0574e94ba9ee7cbf5864a00e7586c133cf2cfee0b1e71"}
Oct 30 22:30:11.411761 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:11.411647    5678 scope.go:122] "RemoveContainer" containerID="e0b2c8c99a7b725e5c0ea317d2bd9569b40e2138e4ce49269cf5c7e79ba9b2a6"
Oct 30 22:30:11.413327 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:11.413296    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-75fb62-845c55c77b-gmd6f" event={"ID":"0e40fbc5-668a-4917-9344-dea69c625d26","Type":"ContainerStarted","Data":"ebb8437912a9e511dc06cdbc8ddfcc72ba2f54ab2250380cf20367dc92b51b9d"}
Oct 30 22:30:11.413482 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:11.413355    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-75fb62-845c55c77b-gmd6f"
Oct 30 22:30:11.571857 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:11.571825    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xq4tp"]
Oct 30 22:30:11.619276 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:11.619248    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xq4tp"]
Oct 30 22:30:11.690651 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:11.690549    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-75fb62-845c55c77b-gmd6f" podStartSLOduration=2.745963521 podStartE2EDuration="3.690530797s" totalImagesPullingTime="944.567276ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:11.688990293 +0000 UTC m=+2389.293297842" watchObservedRunningTime="2026-10-30 22:30:11.690530797 +0000 UTC m=+2389.294838352"
Oct 30 22:30:12.531760 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:12.531731    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2fc93b9-e77d-4306-acec-85ea06f5c4be" path="/var/lib/kubelet/pods/b2fc93b9-e77d-4306-acec-85ea06f5c4be/volumes"
Oct 30 22:30:12.532199 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:12.532185    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfbe138a-ae9f-45e4-9b30-1bf33a80929c" path="/var/lib/kubelet/pods/bfbe138a-ae9f-45e4-9b30-1bf33a80929c/volumes"
Oct 30 22:30:12.532625 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:12.532611    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea5fcd11-31d9-4836-92c3-00952b348844" path="/var/lib/kubelet/pods/ea5fcd11-31d9-4836-92c3-00952b348844/volumes"
Oct 30 22:30:13.516546 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:13.516517    5678 scope.go:122] "RemoveContainer" containerID="02116764a900cfabdc3aa79cbaf8e1c7370de025ac47a3b8d8f72360a4d8c019"
Oct 30 22:30:14.436746 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:14.436713    5678 generic.go:356] "Generic (PLEG): container finished" podID="a94efb34-dd76-4be4-9872-947273ced7c0" containerID="df9bf9b0bf981570c185139ecb68e3e87dc8145adb4951eced5c549858d65953" exitCode=60
Oct 30 22:30:14.437190 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:14.436782    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bc714d-29889990-cxr7b" event={"ID":"a94efb34-dd76-4be4-9872-947273ced7c0","Type":"ContainerDied","Data":"df9bf9b0bf981570c185139ecb68e3e87dc8145adb4951eced5c549858d65953"}
Oct 30 22:30:14.437190 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:14.436843    5678 scope.go:122] "RemoveContainer" containerID="02116764a900cfabdc3aa79cbaf8e1c7370de025ac47a3b8d8f72360a4d8c019"
Oct 30 22:30:14.437365 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:14.437249    5678 scope.go:122] "RemoveContainer" containerID="df9bf9b0bf981570c185139ecb68e3e87dc8145adb4951eced5c549858d65953"
Oct 30 22:30:14.437666 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:30:14.437638    5678 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-bc714d-29889990-cxr7b_odh-ai-gateway-infra(a94efb34-dd76-4be4-9872-947273ced7c0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bc714d-29889990-cxr7b" podUID="a94efb34-dd76-4be4-9872-947273ced7c0"
Oct 30 22:30:15.515797 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:15.515768    5678 scope.go:122] "RemoveContainer" containerID="95c0d0ca6b670b8c42ffaf41b1afcbdd5ecf9afbef96e392481596d9c473b274"
Oct 30 22:30:15.516295 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:15.515853    5678 scope.go:122] "RemoveContainer" containerID="4f8c1aeec04bdafebb85e02ed969995740383a101276d4fb37541d17ece390ff"
Oct 30 22:30:15.563922 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:15.563897    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bc714d-29889990-cxr7b"]
Oct 30 22:30:15.754033 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:15.753994    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bc714d-29889990-cxr7b"
Oct 30 22:30:15.889270 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:15.889241    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a94efb34-dd76-4be4-9872-947273ced7c0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a94efb34-dd76-4be4-9872-947273ced7c0-maas-api-service-ca\") pod \"a94efb34-dd76-4be4-9872-947273ced7c0\" (UID: \"a94efb34-dd76-4be4-9872-947273ced7c0\") "
Oct 30 22:30:15.889468 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:15.889445    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a94efb34-dd76-4be4-9872-947273ced7c0-kube-api-access-qjf4w\" (UniqueName: \"kubernetes.io/projected/a94efb34-dd76-4be4-9872-947273ced7c0-kube-api-access-qjf4w\") pod \"a94efb34-dd76-4be4-9872-947273ced7c0\" (UID: \"a94efb34-dd76-4be4-9872-947273ced7c0\") "
Oct 30 22:30:15.889676 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:15.889652    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a94efb34-dd76-4be4-9872-947273ced7c0-maas-api-service-ca" pod "a94efb34-dd76-4be4-9872-947273ced7c0" (UID: "a94efb34-dd76-4be4-9872-947273ced7c0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:15.889903 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:15.889881    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a94efb34-dd76-4be4-9872-947273ced7c0-maas-api-service-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:15.892010 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:15.891985    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a94efb34-dd76-4be4-9872-947273ced7c0-kube-api-access-qjf4w" pod "a94efb34-dd76-4be4-9872-947273ced7c0" (UID: "a94efb34-dd76-4be4-9872-947273ced7c0"). InnerVolumeSpecName "kube-api-access-qjf4w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:15.991099 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:15.991063    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjf4w\" (UniqueName: \"kubernetes.io/projected/a94efb34-dd76-4be4-9872-947273ced7c0-kube-api-access-qjf4w\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:16.455075 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:16.455048    5678 generic.go:356] "Generic (PLEG): container finished" podID="d0b1525a-699c-4121-94dd-f47d00b08ae7" containerID="fedda9c3c8f04a27a8d3947792c6070d0567cfd80950fe06afe24d81d24fe80b" exitCode=60
Oct 30 22:30:16.455213 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:16.455128    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c42c0a-29889990-z4dwc" event={"ID":"d0b1525a-699c-4121-94dd-f47d00b08ae7","Type":"ContainerDied","Data":"fedda9c3c8f04a27a8d3947792c6070d0567cfd80950fe06afe24d81d24fe80b"}
Oct 30 22:30:16.455277 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:16.455236    5678 scope.go:122] "RemoveContainer" containerID="4f8c1aeec04bdafebb85e02ed969995740383a101276d4fb37541d17ece390ff"
Oct 30 22:30:16.455679 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:16.455657    5678 scope.go:122] "RemoveContainer" containerID="fedda9c3c8f04a27a8d3947792c6070d0567cfd80950fe06afe24d81d24fe80b"
Oct 30 22:30:16.456035 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:30:16.456015    5678 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-c42c0a-29889990-z4dwc_odh-ai-gateway-infra(d0b1525a-699c-4121-94dd-f47d00b08ae7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c42c0a-29889990-z4dwc" podUID="d0b1525a-699c-4121-94dd-f47d00b08ae7"
Oct 30 22:30:16.457213 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:16.457193    5678 generic.go:356] "Generic (PLEG): container finished" podID="e8663b2d-a48a-49a1-a788-7de17d54e62b" containerID="eaab55af1a4b3a5eea9e40f73c340a77987bdbc28cd97980c069656a608e027e" exitCode=60
Oct 30 22:30:16.457302 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:16.457244    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb0b93-29889990-dldtk" event={"ID":"e8663b2d-a48a-49a1-a788-7de17d54e62b","Type":"ContainerDied","Data":"eaab55af1a4b3a5eea9e40f73c340a77987bdbc28cd97980c069656a608e027e"}
Oct 30 22:30:16.457652 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:16.457637    5678 scope.go:122] "RemoveContainer" containerID="eaab55af1a4b3a5eea9e40f73c340a77987bdbc28cd97980c069656a608e027e"
Oct 30 22:30:16.457954 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:30:16.457932    5678 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-fb0b93-29889990-dldtk_odh-ai-gateway-infra(e8663b2d-a48a-49a1-a788-7de17d54e62b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb0b93-29889990-dldtk" podUID="e8663b2d-a48a-49a1-a788-7de17d54e62b"
Oct 30 22:30:16.458593 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:16.458580    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bc714d-29889990-cxr7b"
Oct 30 22:30:16.458655 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:16.458614    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bc714d-29889990-cxr7b" event={"ID":"a94efb34-dd76-4be4-9872-947273ced7c0","Type":"ContainerDied","Data":"5ee8d8b3751c6e31840b0ca198c90bca150b0b479853b46b54bc38f912bb6b11"}
Oct 30 22:30:16.508059 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:16.508040    5678 scope.go:122] "RemoveContainer" containerID="95c0d0ca6b670b8c42ffaf41b1afcbdd5ecf9afbef96e392481596d9c473b274"
Oct 30 22:30:16.522176 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:16.521967    5678 scope.go:122] "RemoveContainer" containerID="df9bf9b0bf981570c185139ecb68e3e87dc8145adb4951eced5c549858d65953"
Oct 30 22:30:16.781968 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:16.781888    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bc714d-29889990-cxr7b"]
Oct 30 22:30:16.786802 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:16.786778    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bc714d-29889990-cxr7b"]
Oct 30 22:30:17.427668 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:17.427630    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-75fb62-845c55c77b-gmd6f"
Oct 30 22:30:17.515846 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:17.515821    5678 scope.go:122] "RemoveContainer" containerID="b33b64f81a78bc08d930e752afd8d204d7a9cc383139b21eb0864b5ac3a70922"
Oct 30 22:30:17.516078 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:17.516060    5678 scope.go:122] "RemoveContainer" containerID="bc9012044bb792f977418aa81796e9d46263075ea556fbd1e88c605b548f686d"
Oct 30 22:30:17.594438 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:17.594394    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c42c0a-29889990-z4dwc"]
Oct 30 22:30:17.688776 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:17.688638    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb0b93-29889990-dldtk"]
Oct 30 22:30:17.764970 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:17.764939    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c42c0a-29889990-z4dwc"
Oct 30 22:30:17.913761 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:17.913724    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0b1525a-699c-4121-94dd-f47d00b08ae7-kube-api-access-mcljh\" (UniqueName: \"kubernetes.io/projected/d0b1525a-699c-4121-94dd-f47d00b08ae7-kube-api-access-mcljh\") pod \"d0b1525a-699c-4121-94dd-f47d00b08ae7\" (UID: \"d0b1525a-699c-4121-94dd-f47d00b08ae7\") "
Oct 30 22:30:17.913946 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:17.913878    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0b1525a-699c-4121-94dd-f47d00b08ae7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0b1525a-699c-4121-94dd-f47d00b08ae7-maas-api-service-ca\") pod \"d0b1525a-699c-4121-94dd-f47d00b08ae7\" (UID: \"d0b1525a-699c-4121-94dd-f47d00b08ae7\") "
Oct 30 22:30:17.914306 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:17.914275    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0b1525a-699c-4121-94dd-f47d00b08ae7-maas-api-service-ca" pod "d0b1525a-699c-4121-94dd-f47d00b08ae7" (UID: "d0b1525a-699c-4121-94dd-f47d00b08ae7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:17.914531 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:17.914509    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0b1525a-699c-4121-94dd-f47d00b08ae7-maas-api-service-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:17.916183 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:17.916159    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0b1525a-699c-4121-94dd-f47d00b08ae7-kube-api-access-mcljh" pod "d0b1525a-699c-4121-94dd-f47d00b08ae7" (UID: "d0b1525a-699c-4121-94dd-f47d00b08ae7"). InnerVolumeSpecName "kube-api-access-mcljh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:17.923865 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:17.923841    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb0b93-29889990-dldtk"
Oct 30 22:30:18.015189 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:18.015110    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8663b2d-a48a-49a1-a788-7de17d54e62b-kube-api-access-zxkzt\" (UniqueName: \"kubernetes.io/projected/e8663b2d-a48a-49a1-a788-7de17d54e62b-kube-api-access-zxkzt\") pod \"e8663b2d-a48a-49a1-a788-7de17d54e62b\" (UID: \"e8663b2d-a48a-49a1-a788-7de17d54e62b\") "
Oct 30 22:30:18.015189 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:18.015158    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8663b2d-a48a-49a1-a788-7de17d54e62b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8663b2d-a48a-49a1-a788-7de17d54e62b-maas-api-service-ca\") pod \"e8663b2d-a48a-49a1-a788-7de17d54e62b\" (UID: \"e8663b2d-a48a-49a1-a788-7de17d54e62b\") "
Oct 30 22:30:18.015636 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:18.015602    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8663b2d-a48a-49a1-a788-7de17d54e62b-maas-api-service-ca" pod "e8663b2d-a48a-49a1-a788-7de17d54e62b" (UID: "e8663b2d-a48a-49a1-a788-7de17d54e62b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:18.015794 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:18.015686    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcljh\" (UniqueName: \"kubernetes.io/projected/d0b1525a-699c-4121-94dd-f47d00b08ae7-kube-api-access-mcljh\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:18.017475 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:18.017442    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8663b2d-a48a-49a1-a788-7de17d54e62b-kube-api-access-zxkzt" pod "e8663b2d-a48a-49a1-a788-7de17d54e62b" (UID: "e8663b2d-a48a-49a1-a788-7de17d54e62b"). InnerVolumeSpecName "kube-api-access-zxkzt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:18.117168 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:18.117134    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxkzt\" (UniqueName: \"kubernetes.io/projected/e8663b2d-a48a-49a1-a788-7de17d54e62b-kube-api-access-zxkzt\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:18.117168 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:18.117161    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8663b2d-a48a-49a1-a788-7de17d54e62b-maas-api-service-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:18.481127 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:18.481101    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c42c0a-29889990-z4dwc"
Oct 30 22:30:18.481320 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:18.481145    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c42c0a-29889990-z4dwc" event={"ID":"d0b1525a-699c-4121-94dd-f47d00b08ae7","Type":"ContainerDied","Data":"0fc0fc770beabc83ca303720f662dd12ed9383d5ae19d080ab045f2611f7d69e"}
Oct 30 22:30:18.481320 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:18.481214    5678 scope.go:122] "RemoveContainer" containerID="fedda9c3c8f04a27a8d3947792c6070d0567cfd80950fe06afe24d81d24fe80b"
Oct 30 22:30:18.482885 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:18.482860    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb0b93-29889990-dldtk"
Oct 30 22:30:18.483023 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:18.482888    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb0b93-29889990-dldtk" event={"ID":"e8663b2d-a48a-49a1-a788-7de17d54e62b","Type":"ContainerDied","Data":"f7d84f772202b745cd2018ee09d653e81f1592e196826e5d4e1ff6b881e786b8"}
Oct 30 22:30:18.497972 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:18.497949    5678 scope.go:122] "RemoveContainer" containerID="eaab55af1a4b3a5eea9e40f73c340a77987bdbc28cd97980c069656a608e027e"
Oct 30 22:30:18.499013 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:18.498980    5678 generic.go:356] "Generic (PLEG): container finished" podID="d886be7a-7208-446a-a3aa-6c478d277578" containerID="87beb5e543d32df04727e3ec460412e2c6e83aeac5afd4518a1a5a532b5ce14e" exitCode=60
Oct 30 22:30:18.499196 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:18.499127    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-99304b-29889990-wn2vq" event={"ID":"d886be7a-7208-446a-a3aa-6c478d277578","Type":"ContainerDied","Data":"87beb5e543d32df04727e3ec460412e2c6e83aeac5afd4518a1a5a532b5ce14e"}
Oct 30 22:30:18.500072 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:18.500047    5678 scope.go:122] "RemoveContainer" containerID="87beb5e543d32df04727e3ec460412e2c6e83aeac5afd4518a1a5a532b5ce14e"
Oct 30 22:30:18.501285 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:30:18.501249    5678 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-99304b-29889990-wn2vq_odh-ai-gateway-infra(d886be7a-7208-446a-a3aa-6c478d277578)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-99304b-29889990-wn2vq" podUID="d886be7a-7208-446a-a3aa-6c478d277578"
Oct 30 22:30:18.504367 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:18.504342    5678 generic.go:356] "Generic (PLEG): container finished" podID="ddca5ef0-e1ab-493c-a3c8-a15e17d91d70" containerID="403203b128f88bdb2974a6b9ef31ad070ad071f516e0d052304b9926a7bd51f2" exitCode=60
Oct 30 22:30:18.504475 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:18.504428    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-779f32-29889990-m7sk5" event={"ID":"ddca5ef0-e1ab-493c-a3c8-a15e17d91d70","Type":"ContainerDied","Data":"403203b128f88bdb2974a6b9ef31ad070ad071f516e0d052304b9926a7bd51f2"}
Oct 30 22:30:18.504760 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:18.504743    5678 scope.go:122] "RemoveContainer" containerID="403203b128f88bdb2974a6b9ef31ad070ad071f516e0d052304b9926a7bd51f2"
Oct 30 22:30:18.504986 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:30:18.504969    5678 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-779f32-29889990-m7sk5_odh-ai-gateway-infra(ddca5ef0-e1ab-493c-a3c8-a15e17d91d70)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-779f32-29889990-m7sk5" podUID="ddca5ef0-e1ab-493c-a3c8-a15e17d91d70"
Oct 30 22:30:18.509734 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:18.509716    5678 scope.go:122] "RemoveContainer" containerID="bc9012044bb792f977418aa81796e9d46263075ea556fbd1e88c605b548f686d"
Oct 30 22:30:18.523331 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:18.523316    5678 scope.go:122] "RemoveContainer" containerID="b33b64f81a78bc08d930e752afd8d204d7a9cc383139b21eb0864b5ac3a70922"
Oct 30 22:30:18.525324 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:18.525301    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a94efb34-dd76-4be4-9872-947273ced7c0" path="/var/lib/kubelet/pods/a94efb34-dd76-4be4-9872-947273ced7c0/volumes"
Oct 30 22:30:18.885635 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:18.884450    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c42c0a-29889990-z4dwc"]
Oct 30 22:30:18.889172 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:18.889136    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c42c0a-29889990-z4dwc"]
Oct 30 22:30:18.974694 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:18.974662    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb0b93-29889990-dldtk"]
Oct 30 22:30:18.979102 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:18.979075    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb0b93-29889990-dldtk"]
Oct 30 22:30:19.658699 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:19.658664    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-99304b-29889990-wn2vq"]
Oct 30 22:30:19.768299 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:19.767144    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-779f32-29889990-m7sk5"]
Oct 30 22:30:19.850582 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:19.850557    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-99304b-29889990-wn2vq"
Oct 30 22:30:19.940002 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:19.939963    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d886be7a-7208-446a-a3aa-6c478d277578-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d886be7a-7208-446a-a3aa-6c478d277578-maas-api-service-ca\") pod \"d886be7a-7208-446a-a3aa-6c478d277578\" (UID: \"d886be7a-7208-446a-a3aa-6c478d277578\") "
Oct 30 22:30:19.940378 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:19.940099    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d886be7a-7208-446a-a3aa-6c478d277578-kube-api-access-wjl9x\" (UniqueName: \"kubernetes.io/projected/d886be7a-7208-446a-a3aa-6c478d277578-kube-api-access-wjl9x\") pod \"d886be7a-7208-446a-a3aa-6c478d277578\" (UID: \"d886be7a-7208-446a-a3aa-6c478d277578\") "
Oct 30 22:30:19.940378 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:19.940288    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d886be7a-7208-446a-a3aa-6c478d277578-maas-api-service-ca" pod "d886be7a-7208-446a-a3aa-6c478d277578" (UID: "d886be7a-7208-446a-a3aa-6c478d277578"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:19.940690 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:19.940669    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d886be7a-7208-446a-a3aa-6c478d277578-maas-api-service-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:19.943447 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:19.943378    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d886be7a-7208-446a-a3aa-6c478d277578-kube-api-access-wjl9x" pod "d886be7a-7208-446a-a3aa-6c478d277578" (UID: "d886be7a-7208-446a-a3aa-6c478d277578"). InnerVolumeSpecName "kube-api-access-wjl9x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:19.963870 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:19.963850    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-779f32-29889990-m7sk5"
Oct 30 22:30:20.041082 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:20.041055    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ddca5ef0-e1ab-493c-a3c8-a15e17d91d70-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddca5ef0-e1ab-493c-a3c8-a15e17d91d70-maas-api-service-ca\") pod \"ddca5ef0-e1ab-493c-a3c8-a15e17d91d70\" (UID: \"ddca5ef0-e1ab-493c-a3c8-a15e17d91d70\") "
Oct 30 22:30:20.041230 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:20.041155    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddca5ef0-e1ab-493c-a3c8-a15e17d91d70-kube-api-access-8pqjz\" (UniqueName: \"kubernetes.io/projected/ddca5ef0-e1ab-493c-a3c8-a15e17d91d70-kube-api-access-8pqjz\") pod \"ddca5ef0-e1ab-493c-a3c8-a15e17d91d70\" (UID: \"ddca5ef0-e1ab-493c-a3c8-a15e17d91d70\") "
Oct 30 22:30:20.041361 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:20.041338    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddca5ef0-e1ab-493c-a3c8-a15e17d91d70-maas-api-service-ca" pod "ddca5ef0-e1ab-493c-a3c8-a15e17d91d70" (UID: "ddca5ef0-e1ab-493c-a3c8-a15e17d91d70"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:20.041765 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:20.041747    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjl9x\" (UniqueName: \"kubernetes.io/projected/d886be7a-7208-446a-a3aa-6c478d277578-kube-api-access-wjl9x\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:20.041839 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:20.041768    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddca5ef0-e1ab-493c-a3c8-a15e17d91d70-maas-api-service-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:20.043699 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:20.043669    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddca5ef0-e1ab-493c-a3c8-a15e17d91d70-kube-api-access-8pqjz" pod "ddca5ef0-e1ab-493c-a3c8-a15e17d91d70" (UID: "ddca5ef0-e1ab-493c-a3c8-a15e17d91d70"). InnerVolumeSpecName "kube-api-access-8pqjz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:20.143204 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:20.143164    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8pqjz\" (UniqueName: \"kubernetes.io/projected/ddca5ef0-e1ab-493c-a3c8-a15e17d91d70-kube-api-access-8pqjz\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:20.523961 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:20.523926    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0b1525a-699c-4121-94dd-f47d00b08ae7" path="/var/lib/kubelet/pods/d0b1525a-699c-4121-94dd-f47d00b08ae7/volumes"
Oct 30 22:30:20.524419 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:20.524389    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8663b2d-a48a-49a1-a788-7de17d54e62b" path="/var/lib/kubelet/pods/e8663b2d-a48a-49a1-a788-7de17d54e62b/volumes"
Oct 30 22:30:20.527312 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:20.527282    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-779f32-29889990-m7sk5"
Oct 30 22:30:20.527447 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:20.527304    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-779f32-29889990-m7sk5" event={"ID":"ddca5ef0-e1ab-493c-a3c8-a15e17d91d70","Type":"ContainerDied","Data":"e1008edc0db52708ac4de3ece53ceb7c7b6df4d6918cc9105de6bad50825e7ff"}
Oct 30 22:30:20.527447 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:20.527376    5678 scope.go:122] "RemoveContainer" containerID="403203b128f88bdb2974a6b9ef31ad070ad071f516e0d052304b9926a7bd51f2"
Oct 30 22:30:20.529119 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:20.529094    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-99304b-29889990-wn2vq" event={"ID":"d886be7a-7208-446a-a3aa-6c478d277578","Type":"ContainerDied","Data":"ac7bb0ddc24427bf3e85d82c2c28338fe7fbb1bb5bc456a58d124cd439276d5f"}
Oct 30 22:30:20.529229 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:20.529133    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-99304b-29889990-wn2vq"
Oct 30 22:30:20.538471 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:20.538454    5678 scope.go:122] "RemoveContainer" containerID="87beb5e543d32df04727e3ec460412e2c6e83aeac5afd4518a1a5a532b5ce14e"
Oct 30 22:30:20.693090 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:20.693058    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-779f32-29889990-m7sk5"]
Oct 30 22:30:20.697521 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:20.697495    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-779f32-29889990-m7sk5"]
Oct 30 22:30:20.801952 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:20.801873    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-99304b-29889990-wn2vq"]
Oct 30 22:30:20.806915 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:20.806882    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-99304b-29889990-wn2vq"]
Oct 30 22:30:22.523890 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:22.523854    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d886be7a-7208-446a-a3aa-6c478d277578" path="/var/lib/kubelet/pods/d886be7a-7208-446a-a3aa-6c478d277578/volumes"
Oct 30 22:30:22.524412 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:22.524383    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ddca5ef0-e1ab-493c-a3c8-a15e17d91d70" path="/var/lib/kubelet/pods/ddca5ef0-e1ab-493c-a3c8-a15e17d91d70/volumes"
Oct 30 22:30:24.073379 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:24.073347    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-7f76c4-79fc59665d-fhwbf"]
Oct 30 22:30:24.073820 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:24.073601    5678 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7f76c4-79fc59665d-fhwbf" podUID="900b0bd8-1f49-4ce5-ad56-16b891688876" containerName="maas-api" containerID="cri-o://c4c08c39286ffe02d5b417dfcefde789b26d1c08a4908d0f9481398565b8f268" gracePeriod=30
Oct 30 22:30:24.357395 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:24.357370    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7f76c4-79fc59665d-fhwbf"
Oct 30 22:30:24.498198 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:24.498113    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/900b0bd8-1f49-4ce5-ad56-16b891688876-kube-api-access-dnmgf\" (UniqueName: \"kubernetes.io/projected/900b0bd8-1f49-4ce5-ad56-16b891688876-kube-api-access-dnmgf\") pod \"900b0bd8-1f49-4ce5-ad56-16b891688876\" (UID: \"900b0bd8-1f49-4ce5-ad56-16b891688876\") "
Oct 30 22:30:24.498361 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:24.498288    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/900b0bd8-1f49-4ce5-ad56-16b891688876-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/900b0bd8-1f49-4ce5-ad56-16b891688876-maas-api-tls\") pod \"900b0bd8-1f49-4ce5-ad56-16b891688876\" (UID: \"900b0bd8-1f49-4ce5-ad56-16b891688876\") "
Oct 30 22:30:24.498361 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:24.498353    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/900b0bd8-1f49-4ce5-ad56-16b891688876-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/900b0bd8-1f49-4ce5-ad56-16b891688876-openshift-service-ca\") pod \"900b0bd8-1f49-4ce5-ad56-16b891688876\" (UID: \"900b0bd8-1f49-4ce5-ad56-16b891688876\") "
Oct 30 22:30:24.498520 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:24.498415    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/900b0bd8-1f49-4ce5-ad56-16b891688876-metrics-certs\" (UniqueName: \"kubernetes.io/secret/900b0bd8-1f49-4ce5-ad56-16b891688876-metrics-certs\") pod \"900b0bd8-1f49-4ce5-ad56-16b891688876\" (UID: \"900b0bd8-1f49-4ce5-ad56-16b891688876\") "
Oct 30 22:30:24.498778 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:24.498746    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/900b0bd8-1f49-4ce5-ad56-16b891688876-openshift-service-ca" pod "900b0bd8-1f49-4ce5-ad56-16b891688876" (UID: "900b0bd8-1f49-4ce5-ad56-16b891688876"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:24.499140 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:24.499114    5678 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/900b0bd8-1f49-4ce5-ad56-16b891688876-openshift-service-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:24.500595 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:24.500567    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/900b0bd8-1f49-4ce5-ad56-16b891688876-kube-api-access-dnmgf" pod "900b0bd8-1f49-4ce5-ad56-16b891688876" (UID: "900b0bd8-1f49-4ce5-ad56-16b891688876"). InnerVolumeSpecName "kube-api-access-dnmgf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:24.501391 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:24.501368    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/900b0bd8-1f49-4ce5-ad56-16b891688876-metrics-certs" pod "900b0bd8-1f49-4ce5-ad56-16b891688876" (UID: "900b0bd8-1f49-4ce5-ad56-16b891688876"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:30:24.501574 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:24.501542    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/900b0bd8-1f49-4ce5-ad56-16b891688876-maas-api-tls" pod "900b0bd8-1f49-4ce5-ad56-16b891688876" (UID: "900b0bd8-1f49-4ce5-ad56-16b891688876"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:30:24.566722 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:24.566688    5678 generic.go:356] "Generic (PLEG): container finished" podID="900b0bd8-1f49-4ce5-ad56-16b891688876" containerID="c4c08c39286ffe02d5b417dfcefde789b26d1c08a4908d0f9481398565b8f268" exitCode=0
Oct 30 22:30:24.567114 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:24.567090    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7f76c4-79fc59665d-fhwbf" event={"ID":"900b0bd8-1f49-4ce5-ad56-16b891688876","Type":"ContainerDied","Data":"c4c08c39286ffe02d5b417dfcefde789b26d1c08a4908d0f9481398565b8f268"}
Oct 30 22:30:24.567336 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:24.567316    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7f76c4-79fc59665d-fhwbf"
Oct 30 22:30:24.567447 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:24.567331    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7f76c4-79fc59665d-fhwbf" event={"ID":"900b0bd8-1f49-4ce5-ad56-16b891688876","Type":"ContainerDied","Data":"6653e4295385e2dbbe7bfa4df6fc25a61eb83b12aed32a0a31ae306a43d12abe"}
Oct 30 22:30:24.567447 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:24.567322    5678 scope.go:122] "RemoveContainer" containerID="c4c08c39286ffe02d5b417dfcefde789b26d1c08a4908d0f9481398565b8f268"
Oct 30 22:30:24.577787 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:24.577763    5678 scope.go:122] "RemoveContainer" containerID="c4c08c39286ffe02d5b417dfcefde789b26d1c08a4908d0f9481398565b8f268"
Oct 30 22:30:24.578164 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:30:24.578131    5678 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4c08c39286ffe02d5b417dfcefde789b26d1c08a4908d0f9481398565b8f268\": container with ID starting with c4c08c39286ffe02d5b417dfcefde789b26d1c08a4908d0f9481398565b8f268 not found: ID does not exist" containerID="c4c08c39286ffe02d5b417dfcefde789b26d1c08a4908d0f9481398565b8f268"
Oct 30 22:30:24.578250 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:24.578188    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4c08c39286ffe02d5b417dfcefde789b26d1c08a4908d0f9481398565b8f268"} err="failed to get container status \"c4c08c39286ffe02d5b417dfcefde789b26d1c08a4908d0f9481398565b8f268\": rpc error: code = NotFound desc = could not find container \"c4c08c39286ffe02d5b417dfcefde789b26d1c08a4908d0f9481398565b8f268\": container with ID starting with c4c08c39286ffe02d5b417dfcefde789b26d1c08a4908d0f9481398565b8f268 not found: ID does not exist"
Oct 30 22:30:24.600219 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:24.600198    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/900b0bd8-1f49-4ce5-ad56-16b891688876-maas-api-tls\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:24.600219 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:24.600220    5678 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/900b0bd8-1f49-4ce5-ad56-16b891688876-metrics-certs\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:24.600374 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:24.600232    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnmgf\" (UniqueName: \"kubernetes.io/projected/900b0bd8-1f49-4ce5-ad56-16b891688876-kube-api-access-dnmgf\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:24.727246 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:24.727215    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-7f76c4-79fc59665d-fhwbf"]
Oct 30 22:30:24.732532 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:24.732499    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-7f76c4-79fc59665d-fhwbf"]
Oct 30 22:30:26.524302 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:26.524267    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="900b0bd8-1f49-4ce5-ad56-16b891688876" path="/var/lib/kubelet/pods/900b0bd8-1f49-4ce5-ad56-16b891688876/volumes"
Oct 30 22:30:28.622384 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:28.622346    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-9f3ea1-openshift-default-5b767dc779-rbqjf"]
Oct 30 22:30:28.622834 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:28.622582    5678 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-9f3ea1-openshift-default-5b767dc779-rbqjf" podUID="6f9052f2-a2a5-47a5-99b8-138d4e59755c" containerName="istio-proxy" containerID="cri-o://f71fdb8fcdf8914f1596b4429501252507c089c3b7623700dbad9f067780248b" gracePeriod=30
Oct 30 22:30:33.889939 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:33.889909    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-9f3ea1-openshift-default-5b767dc779-rbqjf"
Oct 30 22:30:33.912161 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:33.912096    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f9052f2-a2a5-47a5-99b8-138d4e59755c-kube-api-access-hfj6l\" (UniqueName: \"kubernetes.io/projected/6f9052f2-a2a5-47a5-99b8-138d4e59755c-kube-api-access-hfj6l\") pod \"6f9052f2-a2a5-47a5-99b8-138d4e59755c\" (UID: \"6f9052f2-a2a5-47a5-99b8-138d4e59755c\") "
Oct 30 22:30:33.912161 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:33.912143    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f9052f2-a2a5-47a5-99b8-138d4e59755c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f9052f2-a2a5-47a5-99b8-138d4e59755c-istio-envoy\") pod \"6f9052f2-a2a5-47a5-99b8-138d4e59755c\" (UID: \"6f9052f2-a2a5-47a5-99b8-138d4e59755c\") "
Oct 30 22:30:33.912351 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:33.912178    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f9052f2-a2a5-47a5-99b8-138d4e59755c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f9052f2-a2a5-47a5-99b8-138d4e59755c-istio-data\") pod \"6f9052f2-a2a5-47a5-99b8-138d4e59755c\" (UID: \"6f9052f2-a2a5-47a5-99b8-138d4e59755c\") "
Oct 30 22:30:33.912351 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:33.912286    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f9052f2-a2a5-47a5-99b8-138d4e59755c-istio-token\" (UniqueName: \"kubernetes.io/projected/6f9052f2-a2a5-47a5-99b8-138d4e59755c-istio-token\") pod \"6f9052f2-a2a5-47a5-99b8-138d4e59755c\" (UID: \"6f9052f2-a2a5-47a5-99b8-138d4e59755c\") "
Oct 30 22:30:33.912501 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:33.912476    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f9052f2-a2a5-47a5-99b8-138d4e59755c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f9052f2-a2a5-47a5-99b8-138d4e59755c-credential-socket\") pod \"6f9052f2-a2a5-47a5-99b8-138d4e59755c\" (UID: \"6f9052f2-a2a5-47a5-99b8-138d4e59755c\") "
Oct 30 22:30:33.912569 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:33.912513    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f9052f2-a2a5-47a5-99b8-138d4e59755c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f9052f2-a2a5-47a5-99b8-138d4e59755c-workload-socket\") pod \"6f9052f2-a2a5-47a5-99b8-138d4e59755c\" (UID: \"6f9052f2-a2a5-47a5-99b8-138d4e59755c\") "
Oct 30 22:30:33.912569 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:33.912543    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f9052f2-a2a5-47a5-99b8-138d4e59755c-istio-data" pod "6f9052f2-a2a5-47a5-99b8-138d4e59755c" (UID: "6f9052f2-a2a5-47a5-99b8-138d4e59755c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:33.912682 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:33.912574    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f9052f2-a2a5-47a5-99b8-138d4e59755c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f9052f2-a2a5-47a5-99b8-138d4e59755c-istiod-ca-cert\") pod \"6f9052f2-a2a5-47a5-99b8-138d4e59755c\" (UID: \"6f9052f2-a2a5-47a5-99b8-138d4e59755c\") "
Oct 30 22:30:33.912682 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:33.912629    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6f9052f2-a2a5-47a5-99b8-138d4e59755c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f9052f2-a2a5-47a5-99b8-138d4e59755c-istio-podinfo\") pod \"6f9052f2-a2a5-47a5-99b8-138d4e59755c\" (UID: \"6f9052f2-a2a5-47a5-99b8-138d4e59755c\") "
Oct 30 22:30:33.912800 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:33.912703    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f9052f2-a2a5-47a5-99b8-138d4e59755c-credential-socket" pod "6f9052f2-a2a5-47a5-99b8-138d4e59755c" (UID: "6f9052f2-a2a5-47a5-99b8-138d4e59755c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:33.912800 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:33.912735    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f9052f2-a2a5-47a5-99b8-138d4e59755c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f9052f2-a2a5-47a5-99b8-138d4e59755c-workload-certs\") pod \"6f9052f2-a2a5-47a5-99b8-138d4e59755c\" (UID: \"6f9052f2-a2a5-47a5-99b8-138d4e59755c\") "
Oct 30 22:30:33.912979 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:33.912735    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f9052f2-a2a5-47a5-99b8-138d4e59755c-workload-socket" pod "6f9052f2-a2a5-47a5-99b8-138d4e59755c" (UID: "6f9052f2-a2a5-47a5-99b8-138d4e59755c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:33.913051 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:33.912984    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f9052f2-a2a5-47a5-99b8-138d4e59755c-istiod-ca-cert" pod "6f9052f2-a2a5-47a5-99b8-138d4e59755c" (UID: "6f9052f2-a2a5-47a5-99b8-138d4e59755c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:33.913051 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:33.913004    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f9052f2-a2a5-47a5-99b8-138d4e59755c-workload-certs" pod "6f9052f2-a2a5-47a5-99b8-138d4e59755c" (UID: "6f9052f2-a2a5-47a5-99b8-138d4e59755c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:33.913318 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:33.913302    5678 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f9052f2-a2a5-47a5-99b8-138d4e59755c-credential-socket\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:33.913391 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:33.913319    5678 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f9052f2-a2a5-47a5-99b8-138d4e59755c-workload-socket\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:33.913391 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:33.913328    5678 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f9052f2-a2a5-47a5-99b8-138d4e59755c-istiod-ca-cert\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:33.913391 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:33.913338    5678 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f9052f2-a2a5-47a5-99b8-138d4e59755c-workload-certs\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:33.913391 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:33.913346    5678 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f9052f2-a2a5-47a5-99b8-138d4e59755c-istio-data\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:33.914480 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:33.914456    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f9052f2-a2a5-47a5-99b8-138d4e59755c-istio-token" pod "6f9052f2-a2a5-47a5-99b8-138d4e59755c" (UID: "6f9052f2-a2a5-47a5-99b8-138d4e59755c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:33.914784 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:33.914748    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f9052f2-a2a5-47a5-99b8-138d4e59755c-istio-envoy" pod "6f9052f2-a2a5-47a5-99b8-138d4e59755c" (UID: "6f9052f2-a2a5-47a5-99b8-138d4e59755c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:33.915020 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:33.914998    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f9052f2-a2a5-47a5-99b8-138d4e59755c-kube-api-access-hfj6l" pod "6f9052f2-a2a5-47a5-99b8-138d4e59755c" (UID: "6f9052f2-a2a5-47a5-99b8-138d4e59755c"). InnerVolumeSpecName "kube-api-access-hfj6l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:33.915251 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:33.915230    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6f9052f2-a2a5-47a5-99b8-138d4e59755c-istio-podinfo" pod "6f9052f2-a2a5-47a5-99b8-138d4e59755c" (UID: "6f9052f2-a2a5-47a5-99b8-138d4e59755c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:30:34.014910 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:34.014879    5678 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f9052f2-a2a5-47a5-99b8-138d4e59755c-istio-token\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:34.014910 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:34.014908    5678 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f9052f2-a2a5-47a5-99b8-138d4e59755c-istio-podinfo\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:34.014910 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:34.014918    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfj6l\" (UniqueName: \"kubernetes.io/projected/6f9052f2-a2a5-47a5-99b8-138d4e59755c-kube-api-access-hfj6l\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:34.015145 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:34.014927    5678 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f9052f2-a2a5-47a5-99b8-138d4e59755c-istio-envoy\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:34.641500 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:34.641466    5678 generic.go:356] "Generic (PLEG): container finished" podID="6f9052f2-a2a5-47a5-99b8-138d4e59755c" containerID="f71fdb8fcdf8914f1596b4429501252507c089c3b7623700dbad9f067780248b" exitCode=0
Oct 30 22:30:34.641649 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:34.641527    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-9f3ea1-openshift-default-5b767dc779-rbqjf"
Oct 30 22:30:34.641649 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:34.641531    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-9f3ea1-openshift-default-5b767dc779-rbqjf" event={"ID":"6f9052f2-a2a5-47a5-99b8-138d4e59755c","Type":"ContainerDied","Data":"f71fdb8fcdf8914f1596b4429501252507c089c3b7623700dbad9f067780248b"}
Oct 30 22:30:34.641649 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:34.641576    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-9f3ea1-openshift-default-5b767dc779-rbqjf" event={"ID":"6f9052f2-a2a5-47a5-99b8-138d4e59755c","Type":"ContainerDied","Data":"a9e906d652b5a51d0823a549228b5ff3f0234feb1feb40e79e3e64bc8d95873b"}
Oct 30 22:30:34.641649 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:34.641597    5678 scope.go:122] "RemoveContainer" containerID="f71fdb8fcdf8914f1596b4429501252507c089c3b7623700dbad9f067780248b"
Oct 30 22:30:34.651119 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:34.651100    5678 scope.go:122] "RemoveContainer" containerID="2bb759a4dd66467471b7996985692e19c6c378e305e2e6d052dcdf00bd1d7daf"
Oct 30 22:30:34.659858 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:34.659840    5678 scope.go:122] "RemoveContainer" containerID="f71fdb8fcdf8914f1596b4429501252507c089c3b7623700dbad9f067780248b"
Oct 30 22:30:34.660118 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:30:34.660089    5678 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f71fdb8fcdf8914f1596b4429501252507c089c3b7623700dbad9f067780248b\": container with ID starting with f71fdb8fcdf8914f1596b4429501252507c089c3b7623700dbad9f067780248b not found: ID does not exist" containerID="f71fdb8fcdf8914f1596b4429501252507c089c3b7623700dbad9f067780248b"
Oct 30 22:30:34.660215 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:34.660122    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f71fdb8fcdf8914f1596b4429501252507c089c3b7623700dbad9f067780248b"} err="failed to get container status \"f71fdb8fcdf8914f1596b4429501252507c089c3b7623700dbad9f067780248b\": rpc error: code = NotFound desc = could not find container \"f71fdb8fcdf8914f1596b4429501252507c089c3b7623700dbad9f067780248b\": container with ID starting with f71fdb8fcdf8914f1596b4429501252507c089c3b7623700dbad9f067780248b not found: ID does not exist"
Oct 30 22:30:34.660215 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:34.660159    5678 scope.go:122] "RemoveContainer" containerID="2bb759a4dd66467471b7996985692e19c6c378e305e2e6d052dcdf00bd1d7daf"
Oct 30 22:30:34.660446 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:30:34.660422    5678 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bb759a4dd66467471b7996985692e19c6c378e305e2e6d052dcdf00bd1d7daf\": container with ID starting with 2bb759a4dd66467471b7996985692e19c6c378e305e2e6d052dcdf00bd1d7daf not found: ID does not exist" containerID="2bb759a4dd66467471b7996985692e19c6c378e305e2e6d052dcdf00bd1d7daf"
Oct 30 22:30:34.660501 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:34.660452    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bb759a4dd66467471b7996985692e19c6c378e305e2e6d052dcdf00bd1d7daf"} err="failed to get container status \"2bb759a4dd66467471b7996985692e19c6c378e305e2e6d052dcdf00bd1d7daf\": rpc error: code = NotFound desc = could not find container \"2bb759a4dd66467471b7996985692e19c6c378e305e2e6d052dcdf00bd1d7daf\": container with ID starting with 2bb759a4dd66467471b7996985692e19c6c378e305e2e6d052dcdf00bd1d7daf not found: ID does not exist"
Oct 30 22:30:34.803615 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:34.803540    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-9f3ea1-openshift-default-5b767dc779-rbqjf"]
Oct 30 22:30:34.850361 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:34.850334    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-9f3ea1-openshift-default-5b767dc779-rbqjf"]
Oct 30 22:30:35.900494 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:35.900461    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-582593-74c6898768-vr64l"]
Oct 30 22:30:35.901005 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:35.900992    5678 cpu_manager.go:414] "containerMap: removing container" podUID="d886be7a-7208-446a-a3aa-6c478d277578" containerName="cleanup"
Oct 30 22:30:35.901066 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:35.901005    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d886be7a-7208-446a-a3aa-6c478d277578" containerName="cleanup"
Oct 30 22:30:35.901066 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:35.901016    5678 cpu_manager.go:414] "containerMap: removing container" podUID="e8663b2d-a48a-49a1-a788-7de17d54e62b" containerName="cleanup"
Oct 30 22:30:35.901066 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:35.901021    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8663b2d-a48a-49a1-a788-7de17d54e62b" containerName="cleanup"
Oct 30 22:30:35.901066 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:35.901034    5678 cpu_manager.go:414] "containerMap: removing container" podUID="6f9052f2-a2a5-47a5-99b8-138d4e59755c" containerName="istio-proxy"
Oct 30 22:30:35.901066 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:35.901039    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f9052f2-a2a5-47a5-99b8-138d4e59755c" containerName="istio-proxy"
Oct 30 22:30:35.901066 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:35.901047    5678 cpu_manager.go:414] "containerMap: removing container" podUID="900b0bd8-1f49-4ce5-ad56-16b891688876" containerName="maas-api"
Oct 30 22:30:35.901066 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:35.901052    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="900b0bd8-1f49-4ce5-ad56-16b891688876" containerName="maas-api"
Oct 30 22:30:35.901066 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:35.901059    5678 cpu_manager.go:414] "containerMap: removing container" podUID="ddca5ef0-e1ab-493c-a3c8-a15e17d91d70" containerName="cleanup"
Oct 30 22:30:35.901066 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:35.901064    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddca5ef0-e1ab-493c-a3c8-a15e17d91d70" containerName="cleanup"
Oct 30 22:30:35.901321 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:35.901070    5678 cpu_manager.go:414] "containerMap: removing container" podUID="d0b1525a-699c-4121-94dd-f47d00b08ae7" containerName="cleanup"
Oct 30 22:30:35.901321 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:35.901075    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0b1525a-699c-4121-94dd-f47d00b08ae7" containerName="cleanup"
Oct 30 22:30:35.901321 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:35.901082    5678 cpu_manager.go:414] "containerMap: removing container" podUID="ddca5ef0-e1ab-493c-a3c8-a15e17d91d70" containerName="cleanup"
Oct 30 22:30:35.901321 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:35.901086    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddca5ef0-e1ab-493c-a3c8-a15e17d91d70" containerName="cleanup"
Oct 30 22:30:35.901321 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:35.901099    5678 cpu_manager.go:414] "containerMap: removing container" podUID="6f9052f2-a2a5-47a5-99b8-138d4e59755c" containerName="istio-proxy"
Oct 30 22:30:35.901321 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:35.901104    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f9052f2-a2a5-47a5-99b8-138d4e59755c" containerName="istio-proxy"
Oct 30 22:30:35.901321 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:35.901111    5678 cpu_manager.go:414] "containerMap: removing container" podUID="a94efb34-dd76-4be4-9872-947273ced7c0" containerName="cleanup"
Oct 30 22:30:35.901321 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:35.901115    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a94efb34-dd76-4be4-9872-947273ced7c0" containerName="cleanup"
Oct 30 22:30:35.901321 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:35.901122    5678 cpu_manager.go:414] "containerMap: removing container" podUID="b2fc93b9-e77d-4306-acec-85ea06f5c4be" containerName="istio-proxy"
Oct 30 22:30:35.901321 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:35.901128    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2fc93b9-e77d-4306-acec-85ea06f5c4be" containerName="istio-proxy"
Oct 30 22:30:35.901321 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:35.901139    5678 cpu_manager.go:414] "containerMap: removing container" podUID="bfbe138a-ae9f-45e4-9b30-1bf33a80929c" containerName="istio-proxy"
Oct 30 22:30:35.901321 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:35.901144    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfbe138a-ae9f-45e4-9b30-1bf33a80929c" containerName="istio-proxy"
Oct 30 22:30:35.901321 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:35.901152    5678 cpu_manager.go:414] "containerMap: removing container" podUID="d0b1525a-699c-4121-94dd-f47d00b08ae7" containerName="cleanup"
Oct 30 22:30:35.901321 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:35.901157    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0b1525a-699c-4121-94dd-f47d00b08ae7" containerName="cleanup"
Oct 30 22:30:35.901321 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:35.901166    5678 cpu_manager.go:414] "containerMap: removing container" podUID="ea5fcd11-31d9-4836-92c3-00952b348844" containerName="istio-proxy"
Oct 30 22:30:35.901321 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:35.901171    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea5fcd11-31d9-4836-92c3-00952b348844" containerName="istio-proxy"
Oct 30 22:30:35.901321 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:35.901179    5678 cpu_manager.go:414] "containerMap: removing container" podUID="a94efb34-dd76-4be4-9872-947273ced7c0" containerName="cleanup"
Oct 30 22:30:35.901321 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:35.901183    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a94efb34-dd76-4be4-9872-947273ced7c0" containerName="cleanup"
Oct 30 22:30:35.901321 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:35.901191    5678 cpu_manager.go:414] "containerMap: removing container" podUID="a94efb34-dd76-4be4-9872-947273ced7c0" containerName="cleanup"
Oct 30 22:30:35.901321 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:35.901195    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a94efb34-dd76-4be4-9872-947273ced7c0" containerName="cleanup"
Oct 30 22:30:35.901321 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:35.901203    5678 cpu_manager.go:414] "containerMap: removing container" podUID="d886be7a-7208-446a-a3aa-6c478d277578" containerName="cleanup"
Oct 30 22:30:35.901321 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:35.901207    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d886be7a-7208-446a-a3aa-6c478d277578" containerName="cleanup"
Oct 30 22:30:35.901321 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:35.901223    5678 cpu_manager.go:414] "containerMap: removing container" podUID="d0b1525a-699c-4121-94dd-f47d00b08ae7" containerName="cleanup"
Oct 30 22:30:35.901321 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:35.901228    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0b1525a-699c-4121-94dd-f47d00b08ae7" containerName="cleanup"
Oct 30 22:30:35.901321 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:35.901236    5678 cpu_manager.go:414] "containerMap: removing container" podUID="e8663b2d-a48a-49a1-a788-7de17d54e62b" containerName="cleanup"
Oct 30 22:30:35.901321 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:35.901241    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8663b2d-a48a-49a1-a788-7de17d54e62b" containerName="cleanup"
Oct 30 22:30:35.901321 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:35.901315    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfbe138a-ae9f-45e4-9b30-1bf33a80929c" containerName="istio-proxy"
Oct 30 22:30:35.901321 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:35.901324    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0b1525a-699c-4121-94dd-f47d00b08ae7" containerName="cleanup"
Oct 30 22:30:35.901321 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:35.901331    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8663b2d-a48a-49a1-a788-7de17d54e62b" containerName="cleanup"
Oct 30 22:30:35.902290 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:35.901337    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="d886be7a-7208-446a-a3aa-6c478d277578" containerName="cleanup"
Oct 30 22:30:35.902290 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:35.901342    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea5fcd11-31d9-4836-92c3-00952b348844" containerName="istio-proxy"
Oct 30 22:30:35.902290 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:35.901349    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="a94efb34-dd76-4be4-9872-947273ced7c0" containerName="cleanup"
Oct 30 22:30:35.902290 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:35.901354    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="a94efb34-dd76-4be4-9872-947273ced7c0" containerName="cleanup"
Oct 30 22:30:35.902290 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:35.901361    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="d886be7a-7208-446a-a3aa-6c478d277578" containerName="cleanup"
Oct 30 22:30:35.902290 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:35.901366    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8663b2d-a48a-49a1-a788-7de17d54e62b" containerName="cleanup"
Oct 30 22:30:35.902290 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:35.901373    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8663b2d-a48a-49a1-a788-7de17d54e62b" containerName="cleanup"
Oct 30 22:30:35.902290 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:35.901379    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f9052f2-a2a5-47a5-99b8-138d4e59755c" containerName="istio-proxy"
Oct 30 22:30:35.902290 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:35.901383    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="900b0bd8-1f49-4ce5-ad56-16b891688876" containerName="maas-api"
Oct 30 22:30:35.902290 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:35.901389    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddca5ef0-e1ab-493c-a3c8-a15e17d91d70" containerName="cleanup"
Oct 30 22:30:35.902290 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:35.901393    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0b1525a-699c-4121-94dd-f47d00b08ae7" containerName="cleanup"
Oct 30 22:30:35.902290 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:35.901413    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddca5ef0-e1ab-493c-a3c8-a15e17d91d70" containerName="cleanup"
Oct 30 22:30:35.902290 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:35.901422    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2fc93b9-e77d-4306-acec-85ea06f5c4be" containerName="istio-proxy"
Oct 30 22:30:35.906434 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:35.906390    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-582593-74c6898768-vr64l"
Oct 30 22:30:35.936900 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:35.936875    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/09ddef89-ac05-4e76-b7ef-449236df8900-openshift-service-ca\") pod \"maas-api-e2e-inf-a-582593-74c6898768-vr64l\" (UID: \"09ddef89-ac05-4e76-b7ef-449236df8900\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-582593-74c6898768-vr64l"
Oct 30 22:30:35.937002 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:35.936946    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gw7n8\" (UniqueName: \"kubernetes.io/projected/09ddef89-ac05-4e76-b7ef-449236df8900-kube-api-access-gw7n8\") pod \"maas-api-e2e-inf-a-582593-74c6898768-vr64l\" (UID: \"09ddef89-ac05-4e76-b7ef-449236df8900\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-582593-74c6898768-vr64l"
Oct 30 22:30:35.937040 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:35.936999    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/09ddef89-ac05-4e76-b7ef-449236df8900-metrics-certs\") pod \"maas-api-e2e-inf-a-582593-74c6898768-vr64l\" (UID: \"09ddef89-ac05-4e76-b7ef-449236df8900\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-582593-74c6898768-vr64l"
Oct 30 22:30:35.937073 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:35.937053    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/09ddef89-ac05-4e76-b7ef-449236df8900-maas-api-tls\") pod \"maas-api-e2e-inf-a-582593-74c6898768-vr64l\" (UID: \"09ddef89-ac05-4e76-b7ef-449236df8900\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-582593-74c6898768-vr64l"
Oct 30 22:30:35.956343 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:35.956315    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-582593-serving-cert\""
Oct 30 22:30:36.006468 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:36.006438    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-582593-74c6898768-vr64l"]
Oct 30 22:30:36.038242 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:36.038206    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gw7n8\" (UniqueName: \"kubernetes.io/projected/09ddef89-ac05-4e76-b7ef-449236df8900-kube-api-access-gw7n8\") pod \"maas-api-e2e-inf-a-582593-74c6898768-vr64l\" (UID: \"09ddef89-ac05-4e76-b7ef-449236df8900\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-582593-74c6898768-vr64l"
Oct 30 22:30:36.038242 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:36.038245    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/09ddef89-ac05-4e76-b7ef-449236df8900-metrics-certs\") pod \"maas-api-e2e-inf-a-582593-74c6898768-vr64l\" (UID: \"09ddef89-ac05-4e76-b7ef-449236df8900\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-582593-74c6898768-vr64l"
Oct 30 22:30:36.038464 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:36.038295    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/09ddef89-ac05-4e76-b7ef-449236df8900-maas-api-tls\") pod \"maas-api-e2e-inf-a-582593-74c6898768-vr64l\" (UID: \"09ddef89-ac05-4e76-b7ef-449236df8900\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-582593-74c6898768-vr64l"
Oct 30 22:30:36.038464 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:36.038433    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/09ddef89-ac05-4e76-b7ef-449236df8900-openshift-service-ca\") pod \"maas-api-e2e-inf-a-582593-74c6898768-vr64l\" (UID: \"09ddef89-ac05-4e76-b7ef-449236df8900\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-582593-74c6898768-vr64l"
Oct 30 22:30:36.038581 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:30:36.038503    5678 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-582593-serving-cert: secret "maas-api-e2e-inf-a-582593-serving-cert" not found
Oct 30 22:30:36.038581 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:30:36.038568    5678 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/09ddef89-ac05-4e76-b7ef-449236df8900-maas-api-tls podName:09ddef89-ac05-4e76-b7ef-449236df8900 nodeName:}" failed. No retries permitted until 2026-10-30 22:30:36.538550172 +0000 UTC m=+2414.142857720 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/09ddef89-ac05-4e76-b7ef-449236df8900-maas-api-tls") pod "maas-api-e2e-inf-a-582593-74c6898768-vr64l" (UID: "09ddef89-ac05-4e76-b7ef-449236df8900") : secret "maas-api-e2e-inf-a-582593-serving-cert" not found
Oct 30 22:30:36.039079 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:36.039054    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/09ddef89-ac05-4e76-b7ef-449236df8900-openshift-service-ca\") pod \"maas-api-e2e-inf-a-582593-74c6898768-vr64l\" (UID: \"09ddef89-ac05-4e76-b7ef-449236df8900\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-582593-74c6898768-vr64l"
Oct 30 22:30:36.040711 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:36.040689    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/09ddef89-ac05-4e76-b7ef-449236df8900-metrics-certs\") pod \"maas-api-e2e-inf-a-582593-74c6898768-vr64l\" (UID: \"09ddef89-ac05-4e76-b7ef-449236df8900\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-582593-74c6898768-vr64l"
Oct 30 22:30:36.099145 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:36.099113    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gw7n8\" (UniqueName: \"kubernetes.io/projected/09ddef89-ac05-4e76-b7ef-449236df8900-kube-api-access-gw7n8\") pod \"maas-api-e2e-inf-a-582593-74c6898768-vr64l\" (UID: \"09ddef89-ac05-4e76-b7ef-449236df8900\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-582593-74c6898768-vr64l"
Oct 30 22:30:36.524038 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:36.524000    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f9052f2-a2a5-47a5-99b8-138d4e59755c" path="/var/lib/kubelet/pods/6f9052f2-a2a5-47a5-99b8-138d4e59755c/volumes"
Oct 30 22:30:36.544125 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:36.544093    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/09ddef89-ac05-4e76-b7ef-449236df8900-maas-api-tls\") pod \"maas-api-e2e-inf-a-582593-74c6898768-vr64l\" (UID: \"09ddef89-ac05-4e76-b7ef-449236df8900\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-582593-74c6898768-vr64l"
Oct 30 22:30:36.546546 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:36.546518    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/09ddef89-ac05-4e76-b7ef-449236df8900-maas-api-tls\") pod \"maas-api-e2e-inf-a-582593-74c6898768-vr64l\" (UID: \"09ddef89-ac05-4e76-b7ef-449236df8900\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-582593-74c6898768-vr64l"
Oct 30 22:30:36.819386 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:36.819307    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-582593-74c6898768-vr64l"
Oct 30 22:30:37.217228 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:37.217192    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-582593-74c6898768-vr64l"]
Oct 30 22:30:37.223527 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:30:37.223496    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09ddef89_ac05_4e76_b7ef_449236df8900.slice/crio-1d140501fcdd13b9ae73cd3b236a5353b8ecbab6d18d3c9218d1d519698c15f7 WatchSource:0}: Error finding container 1d140501fcdd13b9ae73cd3b236a5353b8ecbab6d18d3c9218d1d519698c15f7: Status 404 returned error can't find the container with id 1d140501fcdd13b9ae73cd3b236a5353b8ecbab6d18d3c9218d1d519698c15f7
Oct 30 22:30:37.662997 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:37.662961    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-582593-74c6898768-vr64l" event={"ID":"09ddef89-ac05-4e76-b7ef-449236df8900","Type":"ContainerStarted","Data":"1d140501fcdd13b9ae73cd3b236a5353b8ecbab6d18d3c9218d1d519698c15f7"}
Oct 30 22:30:38.670730 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:38.670690    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-582593-74c6898768-vr64l" event={"ID":"09ddef89-ac05-4e76-b7ef-449236df8900","Type":"ContainerStarted","Data":"73e1dd2961c02d67fbf1ef227d22d1e5a9c121e012b84b4d7d34bc18655f9fc4"}
Oct 30 22:30:38.671159 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:38.670814    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-582593-74c6898768-vr64l"
Oct 30 22:30:38.785264 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:38.785208    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-582593-74c6898768-vr64l" podStartSLOduration=2.929896972 podStartE2EDuration="3.785194471s" totalImagesPullingTime="855.297499ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:38.784574914 +0000 UTC m=+2416.388882467" watchObservedRunningTime="2026-10-30 22:30:38.785194471 +0000 UTC m=+2416.389502023"
Oct 30 22:30:40.572430 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:30:40.572374    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 22:30:44.682929 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:44.682901    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-582593-74c6898768-vr64l"
Oct 30 22:30:50.759835 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:50.759802    5678 generic.go:356] "Generic (PLEG): container finished" podID="f1691924-5354-4690-8709-6a20bfd81a7e" containerID="df99e50604825f2460a311054363d6eeaafae354297ddbb74bce51ae632392a6" exitCode=0
Oct 30 22:30:50.760259 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:50.759872    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-75fb62-openshift-default-c9cc6b459-4f2fg" event={"ID":"f1691924-5354-4690-8709-6a20bfd81a7e","Type":"ContainerDied","Data":"df99e50604825f2460a311054363d6eeaafae354297ddbb74bce51ae632392a6"}
Oct 30 22:30:50.760259 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:50.760220    5678 scope.go:122] "RemoveContainer" containerID="df99e50604825f2460a311054363d6eeaafae354297ddbb74bce51ae632392a6"
Oct 30 22:30:50.761660 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:50.761637    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:30:50.761727 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:50.761698    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:30:50.761769 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:50.761732    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:30:51.768224 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:51.768184    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-75fb62-openshift-default-c9cc6b459-4f2fg" event={"ID":"f1691924-5354-4690-8709-6a20bfd81a7e","Type":"ContainerStarted","Data":"57eb1b9435314b42a559b796b8611709458066c0d6fe43a95e34c29d1fa6165f"}
Oct 30 22:30:52.358254 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:52.358219    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-75fb62-openshift-default-c9cc6b459-4f2fg"
Oct 30 22:30:52.358454 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:52.358424    5678 patch_prober.go:29] interesting pod/e2e-inf-a-75fb62-openshift-default-c9cc6b459-4f2fg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body=
Oct 30 22:30:52.358515 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:52.358466    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-75fb62-openshift-default-c9cc6b459-4f2fg" podUID="f1691924-5354-4690-8709-6a20bfd81a7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused"
Oct 30 22:30:53.358566 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:53.358534    5678 patch_prober.go:29] interesting pod/e2e-inf-a-75fb62-openshift-default-c9cc6b459-4f2fg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body=
Oct 30 22:30:53.358959 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:53.358596    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-75fb62-openshift-default-c9cc6b459-4f2fg" podUID="f1691924-5354-4690-8709-6a20bfd81a7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused"
Oct 30 22:30:54.357921 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:54.357885    5678 patch_prober.go:29] interesting pod/e2e-inf-a-75fb62-openshift-default-c9cc6b459-4f2fg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body=
Oct 30 22:30:54.358106 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:54.357952    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-75fb62-openshift-default-c9cc6b459-4f2fg" podUID="f1691924-5354-4690-8709-6a20bfd81a7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused"
Oct 30 22:30:54.743633 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:54.743591    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-99304b-6fb7c9447f-8bncq"]
Oct 30 22:30:54.746585 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:54.746487    5678 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-99304b-6fb7c9447f-8bncq" podUID="e0a3b40e-76ff-446a-b81c-4278ea7ce23f" containerName="maas-api" containerID="cri-o://d7ca545f052491a2cd16e5883dd805a3c555f548fad9ccfd0f25e347f1ca00a6" gracePeriod=30
Oct 30 22:30:55.070933 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:55.070903    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-99304b-6fb7c9447f-8bncq"
Oct 30 22:30:55.119906 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:55.119871    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0a3b40e-76ff-446a-b81c-4278ea7ce23f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0a3b40e-76ff-446a-b81c-4278ea7ce23f-metrics-certs\") pod \"e0a3b40e-76ff-446a-b81c-4278ea7ce23f\" (UID: \"e0a3b40e-76ff-446a-b81c-4278ea7ce23f\") "
Oct 30 22:30:55.120076 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:55.119916    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0a3b40e-76ff-446a-b81c-4278ea7ce23f-kube-api-access-kjt9q\" (UniqueName: \"kubernetes.io/projected/e0a3b40e-76ff-446a-b81c-4278ea7ce23f-kube-api-access-kjt9q\") pod \"e0a3b40e-76ff-446a-b81c-4278ea7ce23f\" (UID: \"e0a3b40e-76ff-446a-b81c-4278ea7ce23f\") "
Oct 30 22:30:55.120076 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:55.120020    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0a3b40e-76ff-446a-b81c-4278ea7ce23f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0a3b40e-76ff-446a-b81c-4278ea7ce23f-openshift-service-ca\") pod \"e0a3b40e-76ff-446a-b81c-4278ea7ce23f\" (UID: \"e0a3b40e-76ff-446a-b81c-4278ea7ce23f\") "
Oct 30 22:30:55.120180 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:55.120132    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0a3b40e-76ff-446a-b81c-4278ea7ce23f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0a3b40e-76ff-446a-b81c-4278ea7ce23f-maas-api-tls\") pod \"e0a3b40e-76ff-446a-b81c-4278ea7ce23f\" (UID: \"e0a3b40e-76ff-446a-b81c-4278ea7ce23f\") "
Oct 30 22:30:55.120583 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:55.120525    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0a3b40e-76ff-446a-b81c-4278ea7ce23f-openshift-service-ca" pod "e0a3b40e-76ff-446a-b81c-4278ea7ce23f" (UID: "e0a3b40e-76ff-446a-b81c-4278ea7ce23f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:55.120805 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:55.120781    5678 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0a3b40e-76ff-446a-b81c-4278ea7ce23f-openshift-service-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:55.147601 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:55.147551    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0a3b40e-76ff-446a-b81c-4278ea7ce23f-kube-api-access-kjt9q" pod "e0a3b40e-76ff-446a-b81c-4278ea7ce23f" (UID: "e0a3b40e-76ff-446a-b81c-4278ea7ce23f"). InnerVolumeSpecName "kube-api-access-kjt9q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:55.151612 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:55.151581    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0a3b40e-76ff-446a-b81c-4278ea7ce23f-maas-api-tls" pod "e0a3b40e-76ff-446a-b81c-4278ea7ce23f" (UID: "e0a3b40e-76ff-446a-b81c-4278ea7ce23f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:30:55.152161 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:55.152118    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0a3b40e-76ff-446a-b81c-4278ea7ce23f-metrics-certs" pod "e0a3b40e-76ff-446a-b81c-4278ea7ce23f" (UID: "e0a3b40e-76ff-446a-b81c-4278ea7ce23f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:30:55.227426 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:55.226370    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0a3b40e-76ff-446a-b81c-4278ea7ce23f-maas-api-tls\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:55.227426 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:55.226426    5678 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0a3b40e-76ff-446a-b81c-4278ea7ce23f-metrics-certs\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:55.227426 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:55.226444    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjt9q\" (UniqueName: \"kubernetes.io/projected/e0a3b40e-76ff-446a-b81c-4278ea7ce23f-kube-api-access-kjt9q\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:55.363595 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:55.361081    5678 patch_prober.go:29] interesting pod/e2e-inf-a-75fb62-openshift-default-c9cc6b459-4f2fg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body=
Oct 30 22:30:55.363595 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:55.361136    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-75fb62-openshift-default-c9cc6b459-4f2fg" podUID="f1691924-5354-4690-8709-6a20bfd81a7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused"
Oct 30 22:30:55.797829 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:55.797795    5678 generic.go:356] "Generic (PLEG): container finished" podID="e0a3b40e-76ff-446a-b81c-4278ea7ce23f" containerID="d7ca545f052491a2cd16e5883dd805a3c555f548fad9ccfd0f25e347f1ca00a6" exitCode=0
Oct 30 22:30:55.798296 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:55.797841    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-99304b-6fb7c9447f-8bncq"
Oct 30 22:30:55.798296 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:55.797882    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-99304b-6fb7c9447f-8bncq" event={"ID":"e0a3b40e-76ff-446a-b81c-4278ea7ce23f","Type":"ContainerDied","Data":"d7ca545f052491a2cd16e5883dd805a3c555f548fad9ccfd0f25e347f1ca00a6"}
Oct 30 22:30:55.798296 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:55.797932    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-99304b-6fb7c9447f-8bncq" event={"ID":"e0a3b40e-76ff-446a-b81c-4278ea7ce23f","Type":"ContainerDied","Data":"6e6de717f04f7ae1064b67321ceb085f7be0225cf1767eb62e0339fa7f771d4a"}
Oct 30 22:30:55.798296 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:55.797951    5678 scope.go:122] "RemoveContainer" containerID="d7ca545f052491a2cd16e5883dd805a3c555f548fad9ccfd0f25e347f1ca00a6"
Oct 30 22:30:55.808879 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:55.808853    5678 scope.go:122] "RemoveContainer" containerID="d7ca545f052491a2cd16e5883dd805a3c555f548fad9ccfd0f25e347f1ca00a6"
Oct 30 22:30:55.809117 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:30:55.809100    5678 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7ca545f052491a2cd16e5883dd805a3c555f548fad9ccfd0f25e347f1ca00a6\": container with ID starting with d7ca545f052491a2cd16e5883dd805a3c555f548fad9ccfd0f25e347f1ca00a6 not found: ID does not exist" containerID="d7ca545f052491a2cd16e5883dd805a3c555f548fad9ccfd0f25e347f1ca00a6"
Oct 30 22:30:55.809165 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:55.809126    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7ca545f052491a2cd16e5883dd805a3c555f548fad9ccfd0f25e347f1ca00a6"} err="failed to get container status \"d7ca545f052491a2cd16e5883dd805a3c555f548fad9ccfd0f25e347f1ca00a6\": rpc error: code = NotFound desc = could not find container \"d7ca545f052491a2cd16e5883dd805a3c555f548fad9ccfd0f25e347f1ca00a6\": container with ID starting with d7ca545f052491a2cd16e5883dd805a3c555f548fad9ccfd0f25e347f1ca00a6 not found: ID does not exist"
Oct 30 22:30:55.955855 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:55.955816    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-99304b-6fb7c9447f-8bncq"]
Oct 30 22:30:56.003697 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:56.003660    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-99304b-6fb7c9447f-8bncq"]
Oct 30 22:30:56.358386 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:56.358345    5678 patch_prober.go:29] interesting pod/e2e-inf-a-75fb62-openshift-default-c9cc6b459-4f2fg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body=
Oct 30 22:30:56.358611 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:56.358412    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-75fb62-openshift-default-c9cc6b459-4f2fg" podUID="f1691924-5354-4690-8709-6a20bfd81a7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused"
Oct 30 22:30:56.546926 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:56.543236    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0a3b40e-76ff-446a-b81c-4278ea7ce23f" path="/var/lib/kubelet/pods/e0a3b40e-76ff-446a-b81c-4278ea7ce23f/volumes"
Oct 30 22:30:57.362261 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:57.362232    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-75fb62-openshift-default-c9cc6b459-4f2fg"
Oct 30 22:30:57.362766 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:57.362615    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-75fb62-openshift-default-c9cc6b459-4f2fg"
Oct 30 22:30:57.363228 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:30:57.363206    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-75fb62-openshift-default-c9cc6b459-4f2fg"
Oct 30 22:31:00.662385 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:00.662352    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-65e59f-555d4ff49c-f8ncb"]
Oct 30 22:31:00.662865 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:00.662556    5678 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-65e59f-555d4ff49c-f8ncb" podUID="2834e126-aa61-40fd-86fd-a548211df309" containerName="maas-api" containerID="cri-o://59d1ff72a3743020c6e593e5c8addad669867d00a51cdfc307dda3a0925db52b" gracePeriod=30
Oct 30 22:31:00.835211 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:00.835181    5678 generic.go:356] "Generic (PLEG): container finished" podID="2834e126-aa61-40fd-86fd-a548211df309" containerID="59d1ff72a3743020c6e593e5c8addad669867d00a51cdfc307dda3a0925db52b" exitCode=0
Oct 30 22:31:00.835340 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:00.835260    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-65e59f-555d4ff49c-f8ncb" event={"ID":"2834e126-aa61-40fd-86fd-a548211df309","Type":"ContainerDied","Data":"59d1ff72a3743020c6e593e5c8addad669867d00a51cdfc307dda3a0925db52b"}
Oct 30 22:31:00.937543 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:00.937520    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-65e59f-555d4ff49c-f8ncb"
Oct 30 22:31:01.011906 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:01.011876    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2834e126-aa61-40fd-86fd-a548211df309-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2834e126-aa61-40fd-86fd-a548211df309-metrics-certs\") pod \"2834e126-aa61-40fd-86fd-a548211df309\" (UID: \"2834e126-aa61-40fd-86fd-a548211df309\") "
Oct 30 22:31:01.012088 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:01.011995    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2834e126-aa61-40fd-86fd-a548211df309-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2834e126-aa61-40fd-86fd-a548211df309-maas-api-tls\") pod \"2834e126-aa61-40fd-86fd-a548211df309\" (UID: \"2834e126-aa61-40fd-86fd-a548211df309\") "
Oct 30 22:31:01.012088 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:01.012023    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2834e126-aa61-40fd-86fd-a548211df309-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2834e126-aa61-40fd-86fd-a548211df309-openshift-service-ca\") pod \"2834e126-aa61-40fd-86fd-a548211df309\" (UID: \"2834e126-aa61-40fd-86fd-a548211df309\") "
Oct 30 22:31:01.012215 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:01.012094    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2834e126-aa61-40fd-86fd-a548211df309-kube-api-access-9k765\" (UniqueName: \"kubernetes.io/projected/2834e126-aa61-40fd-86fd-a548211df309-kube-api-access-9k765\") pod \"2834e126-aa61-40fd-86fd-a548211df309\" (UID: \"2834e126-aa61-40fd-86fd-a548211df309\") "
Oct 30 22:31:01.012456 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:01.012392    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2834e126-aa61-40fd-86fd-a548211df309-openshift-service-ca" pod "2834e126-aa61-40fd-86fd-a548211df309" (UID: "2834e126-aa61-40fd-86fd-a548211df309"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:31:01.012604 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:01.012531    5678 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2834e126-aa61-40fd-86fd-a548211df309-openshift-service-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:01.013984 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:01.013958    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2834e126-aa61-40fd-86fd-a548211df309-metrics-certs" pod "2834e126-aa61-40fd-86fd-a548211df309" (UID: "2834e126-aa61-40fd-86fd-a548211df309"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:31:01.014114 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:01.014093    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2834e126-aa61-40fd-86fd-a548211df309-maas-api-tls" pod "2834e126-aa61-40fd-86fd-a548211df309" (UID: "2834e126-aa61-40fd-86fd-a548211df309"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:31:01.014162 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:01.014106    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2834e126-aa61-40fd-86fd-a548211df309-kube-api-access-9k765" pod "2834e126-aa61-40fd-86fd-a548211df309" (UID: "2834e126-aa61-40fd-86fd-a548211df309"). InnerVolumeSpecName "kube-api-access-9k765". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:31:01.113443 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:01.113385    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2834e126-aa61-40fd-86fd-a548211df309-maas-api-tls\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:01.113443 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:01.113432    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9k765\" (UniqueName: \"kubernetes.io/projected/2834e126-aa61-40fd-86fd-a548211df309-kube-api-access-9k765\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:01.113443 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:01.113445    5678 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2834e126-aa61-40fd-86fd-a548211df309-metrics-certs\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:01.843588 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:01.843557    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-65e59f-555d4ff49c-f8ncb"
Oct 30 22:31:01.844035 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:01.843595    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-65e59f-555d4ff49c-f8ncb" event={"ID":"2834e126-aa61-40fd-86fd-a548211df309","Type":"ContainerDied","Data":"c810eb736a9c51f6c01b00c9a0bf4f544bad66215bf2482006200af189f3c84b"}
Oct 30 22:31:01.844035 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:01.843652    5678 scope.go:122] "RemoveContainer" containerID="59d1ff72a3743020c6e593e5c8addad669867d00a51cdfc307dda3a0925db52b"
Oct 30 22:31:02.004452 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:02.004420    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-65e59f-555d4ff49c-f8ncb"]
Oct 30 22:31:02.051204 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:02.051168    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-65e59f-555d4ff49c-f8ncb"]
Oct 30 22:31:02.524366 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:02.524325    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2834e126-aa61-40fd-86fd-a548211df309" path="/var/lib/kubelet/pods/2834e126-aa61-40fd-86fd-a548211df309/volumes"
Oct 30 22:31:10.570570 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:10.570539    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-7jhn9"]
Oct 30 22:31:10.571857 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:10.571806    5678 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-7jhn9" podUID="7fca4f4f-8d98-4ecf-8a41-828fef1ceb56" containerName="istio-proxy" containerID="cri-o://6a7fe0598e9b782a421c6b55f3aeee30480aa96ffeb88b9005eb5d317350a8d9" gracePeriod=30
Oct 30 22:31:10.593272 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:31:10.593234    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 22:31:15.859669 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:15.859644    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-7jhn9"
Oct 30 22:31:15.952071 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:15.951996    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fca4f4f-8d98-4ecf-8a41-828fef1ceb56-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fca4f4f-8d98-4ecf-8a41-828fef1ceb56-credential-socket\") pod \"7fca4f4f-8d98-4ecf-8a41-828fef1ceb56\" (UID: \"7fca4f4f-8d98-4ecf-8a41-828fef1ceb56\") "
Oct 30 22:31:15.952071 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:15.952033    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fca4f4f-8d98-4ecf-8a41-828fef1ceb56-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fca4f4f-8d98-4ecf-8a41-828fef1ceb56-workload-socket\") pod \"7fca4f4f-8d98-4ecf-8a41-828fef1ceb56\" (UID: \"7fca4f4f-8d98-4ecf-8a41-828fef1ceb56\") "
Oct 30 22:31:15.952071 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:15.952063    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fca4f4f-8d98-4ecf-8a41-828fef1ceb56-kube-api-access-8b9td\" (UniqueName: \"kubernetes.io/projected/7fca4f4f-8d98-4ecf-8a41-828fef1ceb56-kube-api-access-8b9td\") pod \"7fca4f4f-8d98-4ecf-8a41-828fef1ceb56\" (UID: \"7fca4f4f-8d98-4ecf-8a41-828fef1ceb56\") "
Oct 30 22:31:15.952362 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:15.952108    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fca4f4f-8d98-4ecf-8a41-828fef1ceb56-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fca4f4f-8d98-4ecf-8a41-828fef1ceb56-workload-certs\") pod \"7fca4f4f-8d98-4ecf-8a41-828fef1ceb56\" (UID: \"7fca4f4f-8d98-4ecf-8a41-828fef1ceb56\") "
Oct 30 22:31:15.952362 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:15.952151    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7fca4f4f-8d98-4ecf-8a41-828fef1ceb56-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fca4f4f-8d98-4ecf-8a41-828fef1ceb56-istiod-ca-cert\") pod \"7fca4f4f-8d98-4ecf-8a41-828fef1ceb56\" (UID: \"7fca4f4f-8d98-4ecf-8a41-828fef1ceb56\") "
Oct 30 22:31:15.952362 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:15.952300    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fca4f4f-8d98-4ecf-8a41-828fef1ceb56-credential-socket" pod "7fca4f4f-8d98-4ecf-8a41-828fef1ceb56" (UID: "7fca4f4f-8d98-4ecf-8a41-828fef1ceb56"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:15.952362 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:15.952345    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7fca4f4f-8d98-4ecf-8a41-828fef1ceb56-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fca4f4f-8d98-4ecf-8a41-828fef1ceb56-istio-podinfo\") pod \"7fca4f4f-8d98-4ecf-8a41-828fef1ceb56\" (UID: \"7fca4f4f-8d98-4ecf-8a41-828fef1ceb56\") "
Oct 30 22:31:15.952610 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:15.952357    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fca4f4f-8d98-4ecf-8a41-828fef1ceb56-workload-socket" pod "7fca4f4f-8d98-4ecf-8a41-828fef1ceb56" (UID: "7fca4f4f-8d98-4ecf-8a41-828fef1ceb56"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:15.952610 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:15.952377    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fca4f4f-8d98-4ecf-8a41-828fef1ceb56-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fca4f4f-8d98-4ecf-8a41-828fef1ceb56-istio-data\") pod \"7fca4f4f-8d98-4ecf-8a41-828fef1ceb56\" (UID: \"7fca4f4f-8d98-4ecf-8a41-828fef1ceb56\") "
Oct 30 22:31:15.952610 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:15.952431    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fca4f4f-8d98-4ecf-8a41-828fef1ceb56-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fca4f4f-8d98-4ecf-8a41-828fef1ceb56-istio-envoy\") pod \"7fca4f4f-8d98-4ecf-8a41-828fef1ceb56\" (UID: \"7fca4f4f-8d98-4ecf-8a41-828fef1ceb56\") "
Oct 30 22:31:15.952610 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:15.952486    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fca4f4f-8d98-4ecf-8a41-828fef1ceb56-istio-token\" (UniqueName: \"kubernetes.io/projected/7fca4f4f-8d98-4ecf-8a41-828fef1ceb56-istio-token\") pod \"7fca4f4f-8d98-4ecf-8a41-828fef1ceb56\" (UID: \"7fca4f4f-8d98-4ecf-8a41-828fef1ceb56\") "
Oct 30 22:31:15.952610 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:15.952514    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fca4f4f-8d98-4ecf-8a41-828fef1ceb56-workload-certs" pod "7fca4f4f-8d98-4ecf-8a41-828fef1ceb56" (UID: "7fca4f4f-8d98-4ecf-8a41-828fef1ceb56"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:15.952912 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:15.952629    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fca4f4f-8d98-4ecf-8a41-828fef1ceb56-istio-data" pod "7fca4f4f-8d98-4ecf-8a41-828fef1ceb56" (UID: "7fca4f4f-8d98-4ecf-8a41-828fef1ceb56"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:15.952912 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:15.952662    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fca4f4f-8d98-4ecf-8a41-828fef1ceb56-istiod-ca-cert" pod "7fca4f4f-8d98-4ecf-8a41-828fef1ceb56" (UID: "7fca4f4f-8d98-4ecf-8a41-828fef1ceb56"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:31:15.953069 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:15.953033    5678 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fca4f4f-8d98-4ecf-8a41-828fef1ceb56-istio-data\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:15.953136 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:15.953078    5678 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fca4f4f-8d98-4ecf-8a41-828fef1ceb56-credential-socket\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:15.953136 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:15.953094    5678 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fca4f4f-8d98-4ecf-8a41-828fef1ceb56-workload-socket\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:15.953136 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:15.953109    5678 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fca4f4f-8d98-4ecf-8a41-828fef1ceb56-workload-certs\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:15.953136 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:15.953122    5678 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fca4f4f-8d98-4ecf-8a41-828fef1ceb56-istiod-ca-cert\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:15.954537 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:15.954514    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7fca4f4f-8d98-4ecf-8a41-828fef1ceb56-istio-podinfo" pod "7fca4f4f-8d98-4ecf-8a41-828fef1ceb56" (UID: "7fca4f4f-8d98-4ecf-8a41-828fef1ceb56"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:31:15.954655 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:15.954635    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fca4f4f-8d98-4ecf-8a41-828fef1ceb56-kube-api-access-8b9td" pod "7fca4f4f-8d98-4ecf-8a41-828fef1ceb56" (UID: "7fca4f4f-8d98-4ecf-8a41-828fef1ceb56"). InnerVolumeSpecName "kube-api-access-8b9td". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:31:15.954716 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:15.954682    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fca4f4f-8d98-4ecf-8a41-828fef1ceb56-istio-envoy" pod "7fca4f4f-8d98-4ecf-8a41-828fef1ceb56" (UID: "7fca4f4f-8d98-4ecf-8a41-828fef1ceb56"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:15.955128 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:15.955107    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fca4f4f-8d98-4ecf-8a41-828fef1ceb56-istio-token" pod "7fca4f4f-8d98-4ecf-8a41-828fef1ceb56" (UID: "7fca4f4f-8d98-4ecf-8a41-828fef1ceb56"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:31:16.005226 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:16.005175    5678 generic.go:356] "Generic (PLEG): container finished" podID="7fca4f4f-8d98-4ecf-8a41-828fef1ceb56" containerID="6a7fe0598e9b782a421c6b55f3aeee30480aa96ffeb88b9005eb5d317350a8d9" exitCode=0
Oct 30 22:31:16.005441 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:16.005234    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-7jhn9"
Oct 30 22:31:16.005441 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:16.005256    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-7jhn9" event={"ID":"7fca4f4f-8d98-4ecf-8a41-828fef1ceb56","Type":"ContainerDied","Data":"6a7fe0598e9b782a421c6b55f3aeee30480aa96ffeb88b9005eb5d317350a8d9"}
Oct 30 22:31:16.005441 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:16.005359    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-7jhn9" event={"ID":"7fca4f4f-8d98-4ecf-8a41-828fef1ceb56","Type":"ContainerDied","Data":"92642760897d363b993768c54972338be7a2f94e3eefb39a6afad09357c977ea"}
Oct 30 22:31:16.005441 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:16.005392    5678 scope.go:122] "RemoveContainer" containerID="6a7fe0598e9b782a421c6b55f3aeee30480aa96ffeb88b9005eb5d317350a8d9"
Oct 30 22:31:16.019944 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:16.019919    5678 scope.go:122] "RemoveContainer" containerID="6a7fe0598e9b782a421c6b55f3aeee30480aa96ffeb88b9005eb5d317350a8d9"
Oct 30 22:31:16.020218 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:31:16.020194    5678 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a7fe0598e9b782a421c6b55f3aeee30480aa96ffeb88b9005eb5d317350a8d9\": container with ID starting with 6a7fe0598e9b782a421c6b55f3aeee30480aa96ffeb88b9005eb5d317350a8d9 not found: ID does not exist" containerID="6a7fe0598e9b782a421c6b55f3aeee30480aa96ffeb88b9005eb5d317350a8d9"
Oct 30 22:31:16.020317 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:16.020227    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a7fe0598e9b782a421c6b55f3aeee30480aa96ffeb88b9005eb5d317350a8d9"} err="failed to get container status \"6a7fe0598e9b782a421c6b55f3aeee30480aa96ffeb88b9005eb5d317350a8d9\": rpc error: code = NotFound desc = could not find container \"6a7fe0598e9b782a421c6b55f3aeee30480aa96ffeb88b9005eb5d317350a8d9\": container with ID starting with 6a7fe0598e9b782a421c6b55f3aeee30480aa96ffeb88b9005eb5d317350a8d9 not found: ID does not exist"
Oct 30 22:31:16.054308 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:16.054276    5678 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fca4f4f-8d98-4ecf-8a41-828fef1ceb56-istio-podinfo\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:16.054308 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:16.054308    5678 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fca4f4f-8d98-4ecf-8a41-828fef1ceb56-istio-envoy\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:16.054565 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:16.054324    5678 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fca4f4f-8d98-4ecf-8a41-828fef1ceb56-istio-token\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:16.054565 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:16.054338    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8b9td\" (UniqueName: \"kubernetes.io/projected/7fca4f4f-8d98-4ecf-8a41-828fef1ceb56-kube-api-access-8b9td\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:16.134333 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:16.134252    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-7jhn9"]
Oct 30 22:31:16.137821 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:16.137795    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-7jhn9"]
Oct 30 22:31:16.533800 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:16.533764    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7fca4f4f-8d98-4ecf-8a41-828fef1ceb56" path="/var/lib/kubelet/pods/7fca4f4f-8d98-4ecf-8a41-828fef1ceb56/volumes"
Oct 30 22:31:19.027971 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:19.027891    5678 generic.go:356] "Generic (PLEG): container finished" podID="e325caa7-d4fb-4a0e-9fe0-ee95742278b3" containerID="bd50cf584c9705bd333a3688197936c0b11b4611c0c209d3b004442718f52b1e" exitCode=0
Oct 30 22:31:19.027971 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:19.027936    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-582593-openshift-default-868f8d8b7-w97qr" event={"ID":"e325caa7-d4fb-4a0e-9fe0-ee95742278b3","Type":"ContainerDied","Data":"bd50cf584c9705bd333a3688197936c0b11b4611c0c209d3b004442718f52b1e"}
Oct 30 22:31:19.028430 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:19.028340    5678 scope.go:122] "RemoveContainer" containerID="bd50cf584c9705bd333a3688197936c0b11b4611c0c209d3b004442718f52b1e"
Oct 30 22:31:19.029762 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:19.029735    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:31:19.029853 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:19.029798    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:31:19.029853 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:19.029831    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:31:20.035591 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:20.035559    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-582593-openshift-default-868f8d8b7-w97qr" event={"ID":"e325caa7-d4fb-4a0e-9fe0-ee95742278b3","Type":"ContainerStarted","Data":"05571cd067d9e4789400a35ab789934ec11012ee436b0285ac8626731519a9eb"}
Oct 30 22:31:20.846427 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:20.842786    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-582593-openshift-default-868f8d8b7-w97qr"
Oct 30 22:31:20.846427 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:20.845504    5678 patch_prober.go:29] interesting pod/e2e-inf-a-582593-openshift-default-868f8d8b7-w97qr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body=
Oct 30 22:31:20.846427 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:20.845554    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-582593-openshift-default-868f8d8b7-w97qr" podUID="e325caa7-d4fb-4a0e-9fe0-ee95742278b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused"
Oct 30 22:31:21.842857 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:21.842813    5678 patch_prober.go:29] interesting pod/e2e-inf-a-582593-openshift-default-868f8d8b7-w97qr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body=
Oct 30 22:31:21.843490 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:21.843452    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-582593-openshift-default-868f8d8b7-w97qr" podUID="e325caa7-d4fb-4a0e-9fe0-ee95742278b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused"
Oct 30 22:31:22.844435 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:22.843817    5678 patch_prober.go:29] interesting pod/e2e-inf-a-582593-openshift-default-868f8d8b7-w97qr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body=
Oct 30 22:31:22.844435 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:22.843892    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-582593-openshift-default-868f8d8b7-w97qr" podUID="e325caa7-d4fb-4a0e-9fe0-ee95742278b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused"
Oct 30 22:31:23.842201 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:23.841762    5678 patch_prober.go:29] interesting pod/e2e-inf-a-582593-openshift-default-868f8d8b7-w97qr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body=
Oct 30 22:31:23.842201 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:23.841838    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-582593-openshift-default-868f8d8b7-w97qr" podUID="e325caa7-d4fb-4a0e-9fe0-ee95742278b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused"
Oct 30 22:31:23.886691 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:23.886661    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-07e9c8-openshift-default-5944f97d49-m76c9"]
Oct 30 22:31:23.887185 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:23.887171    5678 cpu_manager.go:414] "containerMap: removing container" podUID="ddca5ef0-e1ab-493c-a3c8-a15e17d91d70" containerName="cleanup"
Oct 30 22:31:23.887233 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:23.887186    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddca5ef0-e1ab-493c-a3c8-a15e17d91d70" containerName="cleanup"
Oct 30 22:31:23.887233 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:23.887202    5678 cpu_manager.go:414] "containerMap: removing container" podUID="e0a3b40e-76ff-446a-b81c-4278ea7ce23f" containerName="maas-api"
Oct 30 22:31:23.887233 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:23.887207    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0a3b40e-76ff-446a-b81c-4278ea7ce23f" containerName="maas-api"
Oct 30 22:31:23.887358 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:23.887241    5678 cpu_manager.go:414] "containerMap: removing container" podUID="2834e126-aa61-40fd-86fd-a548211df309" containerName="maas-api"
Oct 30 22:31:23.887358 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:23.887248    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2834e126-aa61-40fd-86fd-a548211df309" containerName="maas-api"
Oct 30 22:31:23.887358 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:23.887261    5678 cpu_manager.go:414] "containerMap: removing container" podUID="d886be7a-7208-446a-a3aa-6c478d277578" containerName="cleanup"
Oct 30 22:31:23.887358 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:23.887267    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d886be7a-7208-446a-a3aa-6c478d277578" containerName="cleanup"
Oct 30 22:31:23.887358 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:23.887278    5678 cpu_manager.go:414] "containerMap: removing container" podUID="7fca4f4f-8d98-4ecf-8a41-828fef1ceb56" containerName="istio-proxy"
Oct 30 22:31:23.887358 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:23.887283    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fca4f4f-8d98-4ecf-8a41-828fef1ceb56" containerName="istio-proxy"
Oct 30 22:31:23.887358 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:23.887293    5678 cpu_manager.go:414] "containerMap: removing container" podUID="e8663b2d-a48a-49a1-a788-7de17d54e62b" containerName="cleanup"
Oct 30 22:31:23.887358 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:23.887297    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8663b2d-a48a-49a1-a788-7de17d54e62b" containerName="cleanup"
Oct 30 22:31:23.887648 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:23.887377    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddca5ef0-e1ab-493c-a3c8-a15e17d91d70" containerName="cleanup"
Oct 30 22:31:23.887648 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:23.887385    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0a3b40e-76ff-446a-b81c-4278ea7ce23f" containerName="maas-api"
Oct 30 22:31:23.887648 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:23.887425    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="d886be7a-7208-446a-a3aa-6c478d277578" containerName="cleanup"
Oct 30 22:31:23.887648 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:23.887435    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="a94efb34-dd76-4be4-9872-947273ced7c0" containerName="cleanup"
Oct 30 22:31:23.887648 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:23.887443    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="2834e126-aa61-40fd-86fd-a548211df309" containerName="maas-api"
Oct 30 22:31:23.887648 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:23.887450    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fca4f4f-8d98-4ecf-8a41-828fef1ceb56" containerName="istio-proxy"
Oct 30 22:31:23.887648 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:23.887457    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f9052f2-a2a5-47a5-99b8-138d4e59755c" containerName="istio-proxy"
Oct 30 22:31:23.887648 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:23.887463    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0b1525a-699c-4121-94dd-f47d00b08ae7" containerName="cleanup"
Oct 30 22:31:23.890149 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:23.890127    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-07e9c8-openshift-default-5944f97d49-m76c9"
Oct 30 22:31:23.944026 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:23.943993    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-07e9c8-openshift-default-dockercfg-5q8tm\""
Oct 30 22:31:24.001785 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:24.001741    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8eba014-2735-4189-a982-e33413ce8e4b-istio-envoy\") pod \"e2e-inf-b-07e9c8-openshift-default-5944f97d49-m76c9\" (UID: \"f8eba014-2735-4189-a982-e33413ce8e4b\") " pod="openshift-ingress/e2e-inf-b-07e9c8-openshift-default-5944f97d49-m76c9"
Oct 30 22:31:24.002883 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:24.002857    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8eba014-2735-4189-a982-e33413ce8e4b-istio-data\") pod \"e2e-inf-b-07e9c8-openshift-default-5944f97d49-m76c9\" (UID: \"f8eba014-2735-4189-a982-e33413ce8e4b\") " pod="openshift-ingress/e2e-inf-b-07e9c8-openshift-default-5944f97d49-m76c9"
Oct 30 22:31:24.003153 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:24.003137    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8eba014-2735-4189-a982-e33413ce8e4b-istio-token\") pod \"e2e-inf-b-07e9c8-openshift-default-5944f97d49-m76c9\" (UID: \"f8eba014-2735-4189-a982-e33413ce8e4b\") " pod="openshift-ingress/e2e-inf-b-07e9c8-openshift-default-5944f97d49-m76c9"
Oct 30 22:31:24.003266 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:24.003250    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8eba014-2735-4189-a982-e33413ce8e4b-credential-socket\") pod \"e2e-inf-b-07e9c8-openshift-default-5944f97d49-m76c9\" (UID: \"f8eba014-2735-4189-a982-e33413ce8e4b\") " pod="openshift-ingress/e2e-inf-b-07e9c8-openshift-default-5944f97d49-m76c9"
Oct 30 22:31:24.003389 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:24.003375    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8eba014-2735-4189-a982-e33413ce8e4b-workload-certs\") pod \"e2e-inf-b-07e9c8-openshift-default-5944f97d49-m76c9\" (UID: \"f8eba014-2735-4189-a982-e33413ce8e4b\") " pod="openshift-ingress/e2e-inf-b-07e9c8-openshift-default-5944f97d49-m76c9"
Oct 30 22:31:24.003515 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:24.003501    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8eba014-2735-4189-a982-e33413ce8e4b-istio-podinfo\") pod \"e2e-inf-b-07e9c8-openshift-default-5944f97d49-m76c9\" (UID: \"f8eba014-2735-4189-a982-e33413ce8e4b\") " pod="openshift-ingress/e2e-inf-b-07e9c8-openshift-default-5944f97d49-m76c9"
Oct 30 22:31:24.003751 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:24.003725    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8eba014-2735-4189-a982-e33413ce8e4b-workload-socket\") pod \"e2e-inf-b-07e9c8-openshift-default-5944f97d49-m76c9\" (UID: \"f8eba014-2735-4189-a982-e33413ce8e4b\") " pod="openshift-ingress/e2e-inf-b-07e9c8-openshift-default-5944f97d49-m76c9"
Oct 30 22:31:24.003884 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:24.003781    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qt4l\" (UniqueName: \"kubernetes.io/projected/f8eba014-2735-4189-a982-e33413ce8e4b-kube-api-access-2qt4l\") pod \"e2e-inf-b-07e9c8-openshift-default-5944f97d49-m76c9\" (UID: \"f8eba014-2735-4189-a982-e33413ce8e4b\") " pod="openshift-ingress/e2e-inf-b-07e9c8-openshift-default-5944f97d49-m76c9"
Oct 30 22:31:24.003956 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:24.003888    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8eba014-2735-4189-a982-e33413ce8e4b-istiod-ca-cert\") pod \"e2e-inf-b-07e9c8-openshift-default-5944f97d49-m76c9\" (UID: \"f8eba014-2735-4189-a982-e33413ce8e4b\") " pod="openshift-ingress/e2e-inf-b-07e9c8-openshift-default-5944f97d49-m76c9"
Oct 30 22:31:24.006492 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:24.006470    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-07e9c8-openshift-default-5944f97d49-m76c9"]
Oct 30 22:31:24.104872 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:24.104849    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8eba014-2735-4189-a982-e33413ce8e4b-workload-certs\") pod \"e2e-inf-b-07e9c8-openshift-default-5944f97d49-m76c9\" (UID: \"f8eba014-2735-4189-a982-e33413ce8e4b\") " pod="openshift-ingress/e2e-inf-b-07e9c8-openshift-default-5944f97d49-m76c9"
Oct 30 22:31:24.104976 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:24.104881    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8eba014-2735-4189-a982-e33413ce8e4b-istio-podinfo\") pod \"e2e-inf-b-07e9c8-openshift-default-5944f97d49-m76c9\" (UID: \"f8eba014-2735-4189-a982-e33413ce8e4b\") " pod="openshift-ingress/e2e-inf-b-07e9c8-openshift-default-5944f97d49-m76c9"
Oct 30 22:31:24.105042 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:24.105013    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8eba014-2735-4189-a982-e33413ce8e4b-workload-socket\") pod \"e2e-inf-b-07e9c8-openshift-default-5944f97d49-m76c9\" (UID: \"f8eba014-2735-4189-a982-e33413ce8e4b\") " pod="openshift-ingress/e2e-inf-b-07e9c8-openshift-default-5944f97d49-m76c9"
Oct 30 22:31:24.105104 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:24.105085    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qt4l\" (UniqueName: \"kubernetes.io/projected/f8eba014-2735-4189-a982-e33413ce8e4b-kube-api-access-2qt4l\") pod \"e2e-inf-b-07e9c8-openshift-default-5944f97d49-m76c9\" (UID: \"f8eba014-2735-4189-a982-e33413ce8e4b\") " pod="openshift-ingress/e2e-inf-b-07e9c8-openshift-default-5944f97d49-m76c9"
Oct 30 22:31:24.105160 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:24.105133    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8eba014-2735-4189-a982-e33413ce8e4b-istiod-ca-cert\") pod \"e2e-inf-b-07e9c8-openshift-default-5944f97d49-m76c9\" (UID: \"f8eba014-2735-4189-a982-e33413ce8e4b\") " pod="openshift-ingress/e2e-inf-b-07e9c8-openshift-default-5944f97d49-m76c9"
Oct 30 22:31:24.105223 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:24.105200    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8eba014-2735-4189-a982-e33413ce8e4b-workload-certs\") pod \"e2e-inf-b-07e9c8-openshift-default-5944f97d49-m76c9\" (UID: \"f8eba014-2735-4189-a982-e33413ce8e4b\") " pod="openshift-ingress/e2e-inf-b-07e9c8-openshift-default-5944f97d49-m76c9"
Oct 30 22:31:24.105281 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:24.105234    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8eba014-2735-4189-a982-e33413ce8e4b-istio-envoy\") pod \"e2e-inf-b-07e9c8-openshift-default-5944f97d49-m76c9\" (UID: \"f8eba014-2735-4189-a982-e33413ce8e4b\") " pod="openshift-ingress/e2e-inf-b-07e9c8-openshift-default-5944f97d49-m76c9"
Oct 30 22:31:24.105281 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:24.105272    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8eba014-2735-4189-a982-e33413ce8e4b-istio-data\") pod \"e2e-inf-b-07e9c8-openshift-default-5944f97d49-m76c9\" (UID: \"f8eba014-2735-4189-a982-e33413ce8e4b\") " pod="openshift-ingress/e2e-inf-b-07e9c8-openshift-default-5944f97d49-m76c9"
Oct 30 22:31:24.105449 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:24.105369    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8eba014-2735-4189-a982-e33413ce8e4b-workload-socket\") pod \"e2e-inf-b-07e9c8-openshift-default-5944f97d49-m76c9\" (UID: \"f8eba014-2735-4189-a982-e33413ce8e4b\") " pod="openshift-ingress/e2e-inf-b-07e9c8-openshift-default-5944f97d49-m76c9"
Oct 30 22:31:24.105449 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:24.105436    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8eba014-2735-4189-a982-e33413ce8e4b-istio-token\") pod \"e2e-inf-b-07e9c8-openshift-default-5944f97d49-m76c9\" (UID: \"f8eba014-2735-4189-a982-e33413ce8e4b\") " pod="openshift-ingress/e2e-inf-b-07e9c8-openshift-default-5944f97d49-m76c9"
Oct 30 22:31:24.105564 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:24.105498    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8eba014-2735-4189-a982-e33413ce8e4b-credential-socket\") pod \"e2e-inf-b-07e9c8-openshift-default-5944f97d49-m76c9\" (UID: \"f8eba014-2735-4189-a982-e33413ce8e4b\") " pod="openshift-ingress/e2e-inf-b-07e9c8-openshift-default-5944f97d49-m76c9"
Oct 30 22:31:24.105670 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:24.105648    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8eba014-2735-4189-a982-e33413ce8e4b-istio-data\") pod \"e2e-inf-b-07e9c8-openshift-default-5944f97d49-m76c9\" (UID: \"f8eba014-2735-4189-a982-e33413ce8e4b\") " pod="openshift-ingress/e2e-inf-b-07e9c8-openshift-default-5944f97d49-m76c9"
Oct 30 22:31:24.105787 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:24.105772    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8eba014-2735-4189-a982-e33413ce8e4b-credential-socket\") pod \"e2e-inf-b-07e9c8-openshift-default-5944f97d49-m76c9\" (UID: \"f8eba014-2735-4189-a982-e33413ce8e4b\") " pod="openshift-ingress/e2e-inf-b-07e9c8-openshift-default-5944f97d49-m76c9"
Oct 30 22:31:24.105936 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:24.105914    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8eba014-2735-4189-a982-e33413ce8e4b-istiod-ca-cert\") pod \"e2e-inf-b-07e9c8-openshift-default-5944f97d49-m76c9\" (UID: \"f8eba014-2735-4189-a982-e33413ce8e4b\") " pod="openshift-ingress/e2e-inf-b-07e9c8-openshift-default-5944f97d49-m76c9"
Oct 30 22:31:24.107732 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:24.107696    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8eba014-2735-4189-a982-e33413ce8e4b-istio-envoy\") pod \"e2e-inf-b-07e9c8-openshift-default-5944f97d49-m76c9\" (UID: \"f8eba014-2735-4189-a982-e33413ce8e4b\") " pod="openshift-ingress/e2e-inf-b-07e9c8-openshift-default-5944f97d49-m76c9"
Oct 30 22:31:24.107826 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:24.107774    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8eba014-2735-4189-a982-e33413ce8e4b-istio-podinfo\") pod \"e2e-inf-b-07e9c8-openshift-default-5944f97d49-m76c9\" (UID: \"f8eba014-2735-4189-a982-e33413ce8e4b\") " pod="openshift-ingress/e2e-inf-b-07e9c8-openshift-default-5944f97d49-m76c9"
Oct 30 22:31:24.164322 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:24.164293    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8eba014-2735-4189-a982-e33413ce8e4b-istio-token\") pod \"e2e-inf-b-07e9c8-openshift-default-5944f97d49-m76c9\" (UID: \"f8eba014-2735-4189-a982-e33413ce8e4b\") " pod="openshift-ingress/e2e-inf-b-07e9c8-openshift-default-5944f97d49-m76c9"
Oct 30 22:31:24.164800 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:24.164778    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qt4l\" (UniqueName: \"kubernetes.io/projected/f8eba014-2735-4189-a982-e33413ce8e4b-kube-api-access-2qt4l\") pod \"e2e-inf-b-07e9c8-openshift-default-5944f97d49-m76c9\" (UID: \"f8eba014-2735-4189-a982-e33413ce8e4b\") " pod="openshift-ingress/e2e-inf-b-07e9c8-openshift-default-5944f97d49-m76c9"
Oct 30 22:31:24.210264 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:24.210231    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-07e9c8-openshift-default-5944f97d49-m76c9"
Oct 30 22:31:24.690199 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:24.690165    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-07e9c8-openshift-default-5944f97d49-m76c9"]
Oct 30 22:31:24.696915 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:31:24.696886    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8eba014_2735_4189_a982_e33413ce8e4b.slice/crio-283b845b3778e9357d462d2cecf8e80b7817bff321185ed875cdd807b151ca66 WatchSource:0}: Error finding container 283b845b3778e9357d462d2cecf8e80b7817bff321185ed875cdd807b151ca66: Status 404 returned error can't find the container with id 283b845b3778e9357d462d2cecf8e80b7817bff321185ed875cdd807b151ca66
Oct 30 22:31:24.699494 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:24.699466    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:31:24.699578 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:24.699528    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:31:24.699578 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:24.699556    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:31:24.845425 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:24.842819    5678 patch_prober.go:29] interesting pod/e2e-inf-a-582593-openshift-default-868f8d8b7-w97qr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body=
Oct 30 22:31:24.845425 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:24.842877    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-582593-openshift-default-868f8d8b7-w97qr" podUID="e325caa7-d4fb-4a0e-9fe0-ee95742278b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused"
Oct 30 22:31:25.094111 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:25.094022    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-07e9c8-openshift-default-5944f97d49-m76c9" event={"ID":"f8eba014-2735-4189-a982-e33413ce8e4b","Type":"ContainerStarted","Data":"a5e150eb8bd120fca6311562b9ea5e1f23c79a3670a3aa5f575f15df9856d9a1"}
Oct 30 22:31:25.094111 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:25.094085    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-07e9c8-openshift-default-5944f97d49-m76c9" event={"ID":"f8eba014-2735-4189-a982-e33413ce8e4b","Type":"ContainerStarted","Data":"283b845b3778e9357d462d2cecf8e80b7817bff321185ed875cdd807b151ca66"}
Oct 30 22:31:25.210670 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:25.210638    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-07e9c8-openshift-default-5944f97d49-m76c9"
Oct 30 22:31:25.216294 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:25.216271    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-07e9c8-openshift-default-5944f97d49-m76c9"
Oct 30 22:31:25.220545 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:25.220499    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-07e9c8-openshift-default-5944f97d49-m76c9" podStartSLOduration=2.2204862419999998 podStartE2EDuration="2.220486242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:31:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:31:25.218765393 +0000 UTC m=+2462.823072947" watchObservedRunningTime="2026-10-30 22:31:25.220486242 +0000 UTC m=+2462.824793794"
Oct 30 22:31:25.847937 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:25.847904    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-582593-openshift-default-868f8d8b7-w97qr"
Oct 30 22:31:25.848342 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:25.848314    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-582593-openshift-default-868f8d8b7-w97qr"
Oct 30 22:31:25.849250 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:25.849230    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-582593-openshift-default-868f8d8b7-w97qr"
Oct 30 22:31:26.102944 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:26.102858    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-07e9c8-openshift-default-5944f97d49-m76c9"
Oct 30 22:31:26.104035 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:26.104010    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-07e9c8-openshift-default-5944f97d49-m76c9"
Oct 30 22:31:31.687310 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:31.687281    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-07e9c8-5f45d5ff75-z224x"]
Oct 30 22:31:31.690907 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:31.690875    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-07e9c8-5f45d5ff75-z224x"
Oct 30 22:31:31.745698 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:31.745647    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-07e9c8-dockercfg-ddhmr\""
Oct 30 22:31:31.746479 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:31.746314    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-07e9c8\""
Oct 30 22:31:31.773617 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:31.756454    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-07e9c8-5f45d5ff75-z224x"]
Oct 30 22:31:31.823314 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:31.823264    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47873f24-8e29-439b-8017-776ece323a1c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-07e9c8-5f45d5ff75-z224x\" (UID: \"47873f24-8e29-439b-8017-776ece323a1c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-07e9c8-5f45d5ff75-z224x"
Oct 30 22:31:31.823566 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:31.823542    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlrch\" (UniqueName: \"kubernetes.io/projected/47873f24-8e29-439b-8017-776ece323a1c-kube-api-access-rlrch\") pod \"payload-pre-processing-e2e-inf-b-07e9c8-5f45d5ff75-z224x\" (UID: \"47873f24-8e29-439b-8017-776ece323a1c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-07e9c8-5f45d5ff75-z224x"
Oct 30 22:31:31.838073 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:31.838042    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-07e9c8-5bd8cdfbff-h8vtx"]
Oct 30 22:31:31.841309 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:31.841287    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-07e9c8-5bd8cdfbff-h8vtx"
Oct 30 22:31:31.891638 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:31.891610    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-07e9c8-5bd8cdfbff-h8vtx"]
Oct 30 22:31:31.924690 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:31.924651    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlrch\" (UniqueName: \"kubernetes.io/projected/47873f24-8e29-439b-8017-776ece323a1c-kube-api-access-rlrch\") pod \"payload-pre-processing-e2e-inf-b-07e9c8-5f45d5ff75-z224x\" (UID: \"47873f24-8e29-439b-8017-776ece323a1c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-07e9c8-5f45d5ff75-z224x"
Oct 30 22:31:31.924887 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:31.924762    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47873f24-8e29-439b-8017-776ece323a1c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-07e9c8-5f45d5ff75-z224x\" (UID: \"47873f24-8e29-439b-8017-776ece323a1c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-07e9c8-5f45d5ff75-z224x"
Oct 30 22:31:31.924887 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:31.924837    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab8b95b5-8eca-405d-bfa9-195e5b4f68f7-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-07e9c8-5bd8cdfbff-h8vtx\" (UID: \"ab8b95b5-8eca-405d-bfa9-195e5b4f68f7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-07e9c8-5bd8cdfbff-h8vtx"
Oct 30 22:31:31.924887 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:31.924866    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcwh4\" (UniqueName: \"kubernetes.io/projected/ab8b95b5-8eca-405d-bfa9-195e5b4f68f7-kube-api-access-fcwh4\") pod \"payload-processing-e2e-inf-b-07e9c8-5bd8cdfbff-h8vtx\" (UID: \"ab8b95b5-8eca-405d-bfa9-195e5b4f68f7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-07e9c8-5bd8cdfbff-h8vtx"
Oct 30 22:31:31.925481 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:31.925460    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47873f24-8e29-439b-8017-776ece323a1c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-07e9c8-5f45d5ff75-z224x\" (UID: \"47873f24-8e29-439b-8017-776ece323a1c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-07e9c8-5f45d5ff75-z224x"
Oct 30 22:31:31.984589 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:31.984510    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlrch\" (UniqueName: \"kubernetes.io/projected/47873f24-8e29-439b-8017-776ece323a1c-kube-api-access-rlrch\") pod \"payload-pre-processing-e2e-inf-b-07e9c8-5f45d5ff75-z224x\" (UID: \"47873f24-8e29-439b-8017-776ece323a1c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-07e9c8-5f45d5ff75-z224x"
Oct 30 22:31:32.004243 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:32.004216    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-07e9c8-5f45d5ff75-z224x"
Oct 30 22:31:32.026214 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:32.026183    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab8b95b5-8eca-405d-bfa9-195e5b4f68f7-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-07e9c8-5bd8cdfbff-h8vtx\" (UID: \"ab8b95b5-8eca-405d-bfa9-195e5b4f68f7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-07e9c8-5bd8cdfbff-h8vtx"
Oct 30 22:31:32.026338 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:32.026226    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcwh4\" (UniqueName: \"kubernetes.io/projected/ab8b95b5-8eca-405d-bfa9-195e5b4f68f7-kube-api-access-fcwh4\") pod \"payload-processing-e2e-inf-b-07e9c8-5bd8cdfbff-h8vtx\" (UID: \"ab8b95b5-8eca-405d-bfa9-195e5b4f68f7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-07e9c8-5bd8cdfbff-h8vtx"
Oct 30 22:31:32.026880 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:32.026859    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab8b95b5-8eca-405d-bfa9-195e5b4f68f7-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-07e9c8-5bd8cdfbff-h8vtx\" (UID: \"ab8b95b5-8eca-405d-bfa9-195e5b4f68f7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-07e9c8-5bd8cdfbff-h8vtx"
Oct 30 22:31:32.085428 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:32.085361    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcwh4\" (UniqueName: \"kubernetes.io/projected/ab8b95b5-8eca-405d-bfa9-195e5b4f68f7-kube-api-access-fcwh4\") pod \"payload-processing-e2e-inf-b-07e9c8-5bd8cdfbff-h8vtx\" (UID: \"ab8b95b5-8eca-405d-bfa9-195e5b4f68f7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-07e9c8-5bd8cdfbff-h8vtx"
Oct 30 22:31:32.154847 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:32.154812    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-07e9c8-5bd8cdfbff-h8vtx"
Oct 30 22:31:32.390865 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:32.390836    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-07e9c8-5f45d5ff75-z224x"]
Oct 30 22:31:32.397728 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:31:32.397702    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47873f24_8e29_439b_8017_776ece323a1c.slice/crio-7aa02f39ac9bfdff442305f301ff58ae88afd855ddeb2f38b9bf40e628583691 WatchSource:0}: Error finding container 7aa02f39ac9bfdff442305f301ff58ae88afd855ddeb2f38b9bf40e628583691: Status 404 returned error can't find the container with id 7aa02f39ac9bfdff442305f301ff58ae88afd855ddeb2f38b9bf40e628583691
Oct 30 22:31:32.539821 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:32.539796    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-07e9c8-5bd8cdfbff-h8vtx"]
Oct 30 22:31:32.545896 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:31:32.545873    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab8b95b5_8eca_405d_bfa9_195e5b4f68f7.slice/crio-39309ac761d1d7d22dc6d5ddfbb2cce1a8dfc9880b89d74e20c69957fe920fff WatchSource:0}: Error finding container 39309ac761d1d7d22dc6d5ddfbb2cce1a8dfc9880b89d74e20c69957fe920fff: Status 404 returned error can't find the container with id 39309ac761d1d7d22dc6d5ddfbb2cce1a8dfc9880b89d74e20c69957fe920fff
Oct 30 22:31:33.146163 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:33.146123    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-07e9c8-5bd8cdfbff-h8vtx" event={"ID":"ab8b95b5-8eca-405d-bfa9-195e5b4f68f7","Type":"ContainerStarted","Data":"c23c11290c0bf58481911bdb94015c88696afc359ce77e5cdd6fa848dc598a3c"}
Oct 30 22:31:33.146587 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:33.146194    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-07e9c8-5bd8cdfbff-h8vtx"
Oct 30 22:31:33.146587 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:33.146220    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-07e9c8-5bd8cdfbff-h8vtx" event={"ID":"ab8b95b5-8eca-405d-bfa9-195e5b4f68f7","Type":"ContainerStarted","Data":"39309ac761d1d7d22dc6d5ddfbb2cce1a8dfc9880b89d74e20c69957fe920fff"}
Oct 30 22:31:33.147743 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:33.147714    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-07e9c8-5f45d5ff75-z224x" event={"ID":"47873f24-8e29-439b-8017-776ece323a1c","Type":"ContainerStarted","Data":"4fd044d0bf9c012d99f4d112c479b74f487d514963982abe3a65523687363140"}
Oct 30 22:31:33.147848 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:33.147768    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-07e9c8-5f45d5ff75-z224x" event={"ID":"47873f24-8e29-439b-8017-776ece323a1c","Type":"ContainerStarted","Data":"7aa02f39ac9bfdff442305f301ff58ae88afd855ddeb2f38b9bf40e628583691"}
Oct 30 22:31:33.147848 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:33.147797    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-07e9c8-5f45d5ff75-z224x"
Oct 30 22:31:33.260231 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:33.260173    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-07e9c8-5bd8cdfbff-h8vtx" podStartSLOduration=2.260160049 podStartE2EDuration="2.260160049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:31:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:31:33.258448498 +0000 UTC m=+2470.862756051" watchObservedRunningTime="2026-10-30 22:31:33.260160049 +0000 UTC m=+2470.864467603"
Oct 30 22:31:33.378348 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:33.378231    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-07e9c8-5f45d5ff75-z224x" podStartSLOduration=2.378211688 podStartE2EDuration="2.378211688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:31:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:31:33.376312209 +0000 UTC m=+2470.980619763" watchObservedRunningTime="2026-10-30 22:31:33.378211688 +0000 UTC m=+2470.982519242"
Oct 30 22:31:40.572354 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:31:40.572319    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 22:31:41.125428 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:41.108119    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-5686cb-ff9ffd898-k5pgw"]
Oct 30 22:31:41.125428 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:41.108372    5678 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5686cb-ff9ffd898-k5pgw" podUID="552ec7ea-ea71-43fa-8129-9829f804fab6" containerName="maas-api" containerID="cri-o://d27d2ea771ec8e2db7943ef34f4faf9678ebded549da1747a28daf41658e5222" gracePeriod=30
Oct 30 22:31:41.408870 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:41.408845    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5686cb-ff9ffd898-k5pgw"
Oct 30 22:31:41.541439 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:41.541396    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/552ec7ea-ea71-43fa-8129-9829f804fab6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/552ec7ea-ea71-43fa-8129-9829f804fab6-metrics-certs\") pod \"552ec7ea-ea71-43fa-8129-9829f804fab6\" (UID: \"552ec7ea-ea71-43fa-8129-9829f804fab6\") "
Oct 30 22:31:41.541621 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:41.541468    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/552ec7ea-ea71-43fa-8129-9829f804fab6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/552ec7ea-ea71-43fa-8129-9829f804fab6-maas-api-tls\") pod \"552ec7ea-ea71-43fa-8129-9829f804fab6\" (UID: \"552ec7ea-ea71-43fa-8129-9829f804fab6\") "
Oct 30 22:31:41.541700 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:41.541637    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/552ec7ea-ea71-43fa-8129-9829f804fab6-kube-api-access-4zslp\" (UniqueName: \"kubernetes.io/projected/552ec7ea-ea71-43fa-8129-9829f804fab6-kube-api-access-4zslp\") pod \"552ec7ea-ea71-43fa-8129-9829f804fab6\" (UID: \"552ec7ea-ea71-43fa-8129-9829f804fab6\") "
Oct 30 22:31:41.541700 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:41.541679    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/552ec7ea-ea71-43fa-8129-9829f804fab6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/552ec7ea-ea71-43fa-8129-9829f804fab6-openshift-service-ca\") pod \"552ec7ea-ea71-43fa-8129-9829f804fab6\" (UID: \"552ec7ea-ea71-43fa-8129-9829f804fab6\") "
Oct 30 22:31:41.542136 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:41.542105    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/552ec7ea-ea71-43fa-8129-9829f804fab6-openshift-service-ca" pod "552ec7ea-ea71-43fa-8129-9829f804fab6" (UID: "552ec7ea-ea71-43fa-8129-9829f804fab6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:31:41.543641 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:41.543620    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/552ec7ea-ea71-43fa-8129-9829f804fab6-metrics-certs" pod "552ec7ea-ea71-43fa-8129-9829f804fab6" (UID: "552ec7ea-ea71-43fa-8129-9829f804fab6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:31:41.544013 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:41.543995    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/552ec7ea-ea71-43fa-8129-9829f804fab6-maas-api-tls" pod "552ec7ea-ea71-43fa-8129-9829f804fab6" (UID: "552ec7ea-ea71-43fa-8129-9829f804fab6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:31:41.544125 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:41.544111    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/552ec7ea-ea71-43fa-8129-9829f804fab6-kube-api-access-4zslp" pod "552ec7ea-ea71-43fa-8129-9829f804fab6" (UID: "552ec7ea-ea71-43fa-8129-9829f804fab6"). InnerVolumeSpecName "kube-api-access-4zslp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:31:41.643601 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:41.643347    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zslp\" (UniqueName: \"kubernetes.io/projected/552ec7ea-ea71-43fa-8129-9829f804fab6-kube-api-access-4zslp\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:41.643601 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:41.643385    5678 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/552ec7ea-ea71-43fa-8129-9829f804fab6-openshift-service-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:41.643601 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:41.643418    5678 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/552ec7ea-ea71-43fa-8129-9829f804fab6-metrics-certs\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:41.643601 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:41.643432    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/552ec7ea-ea71-43fa-8129-9829f804fab6-maas-api-tls\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:42.218577 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:42.218539    5678 generic.go:356] "Generic (PLEG): container finished" podID="552ec7ea-ea71-43fa-8129-9829f804fab6" containerID="d27d2ea771ec8e2db7943ef34f4faf9678ebded549da1747a28daf41658e5222" exitCode=0
Oct 30 22:31:42.218758 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:42.218594    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5686cb-ff9ffd898-k5pgw"
Oct 30 22:31:42.218758 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:42.218631    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5686cb-ff9ffd898-k5pgw" event={"ID":"552ec7ea-ea71-43fa-8129-9829f804fab6","Type":"ContainerDied","Data":"d27d2ea771ec8e2db7943ef34f4faf9678ebded549da1747a28daf41658e5222"}
Oct 30 22:31:42.218758 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:42.218695    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5686cb-ff9ffd898-k5pgw" event={"ID":"552ec7ea-ea71-43fa-8129-9829f804fab6","Type":"ContainerDied","Data":"c3ef68c56f5cff102743d5fb864d4b082b237b75f3d414d747d80c3a672cf63e"}
Oct 30 22:31:42.218758 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:42.218717    5678 scope.go:122] "RemoveContainer" containerID="d27d2ea771ec8e2db7943ef34f4faf9678ebded549da1747a28daf41658e5222"
Oct 30 22:31:42.229499 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:42.229478    5678 scope.go:122] "RemoveContainer" containerID="d27d2ea771ec8e2db7943ef34f4faf9678ebded549da1747a28daf41658e5222"
Oct 30 22:31:42.229756 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:31:42.229738    5678 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d27d2ea771ec8e2db7943ef34f4faf9678ebded549da1747a28daf41658e5222\": container with ID starting with d27d2ea771ec8e2db7943ef34f4faf9678ebded549da1747a28daf41658e5222 not found: ID does not exist" containerID="d27d2ea771ec8e2db7943ef34f4faf9678ebded549da1747a28daf41658e5222"
Oct 30 22:31:42.229807 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:42.229766    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d27d2ea771ec8e2db7943ef34f4faf9678ebded549da1747a28daf41658e5222"} err="failed to get container status \"d27d2ea771ec8e2db7943ef34f4faf9678ebded549da1747a28daf41658e5222\": rpc error: code = NotFound desc = could not find container \"d27d2ea771ec8e2db7943ef34f4faf9678ebded549da1747a28daf41658e5222\": container with ID starting with d27d2ea771ec8e2db7943ef34f4faf9678ebded549da1747a28daf41658e5222 not found: ID does not exist"
Oct 30 22:31:42.376302 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:42.376265    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-5686cb-ff9ffd898-k5pgw"]
Oct 30 22:31:42.380418 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:42.380381    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-5686cb-ff9ffd898-k5pgw"]
Oct 30 22:31:42.524678 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:42.524603    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="552ec7ea-ea71-43fa-8129-9829f804fab6" path="/var/lib/kubelet/pods/552ec7ea-ea71-43fa-8129-9829f804fab6/volumes"
Oct 30 22:31:44.160344 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:44.160310    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-07e9c8-5bd8cdfbff-h8vtx"
Oct 30 22:31:44.160948 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:44.160417    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-07e9c8-5f45d5ff75-z224x"
Oct 30 22:31:47.399315 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:47.399283    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-506964-openshift-default-5bfdbc5849-4qwwq"]
Oct 30 22:31:47.400085 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:47.400067    5678 cpu_manager.go:414] "containerMap: removing container" podUID="552ec7ea-ea71-43fa-8129-9829f804fab6" containerName="maas-api"
Oct 30 22:31:47.400145 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:47.400088    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="552ec7ea-ea71-43fa-8129-9829f804fab6" containerName="maas-api"
Oct 30 22:31:47.400221 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:47.400210    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="552ec7ea-ea71-43fa-8129-9829f804fab6" containerName="maas-api"
Oct 30 22:31:47.403778 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:47.403757    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-506964-openshift-default-5bfdbc5849-4qwwq"
Oct 30 22:31:47.456424 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:47.455580    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-506964-openshift-default-dockercfg-r2jk8\""
Oct 30 22:31:47.510685 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:47.510650    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-506964-openshift-default-5bfdbc5849-4qwwq"]
Oct 30 22:31:47.574139 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:47.572682    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64a0ba20-2130-4df1-b147-28a52a7d0e62-istio-token\") pod \"e2e-inf-b-506964-openshift-default-5bfdbc5849-4qwwq\" (UID: \"64a0ba20-2130-4df1-b147-28a52a7d0e62\") " pod="openshift-ingress/e2e-inf-b-506964-openshift-default-5bfdbc5849-4qwwq"
Oct 30 22:31:47.574139 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:47.572751    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64a0ba20-2130-4df1-b147-28a52a7d0e62-istio-envoy\") pod \"e2e-inf-b-506964-openshift-default-5bfdbc5849-4qwwq\" (UID: \"64a0ba20-2130-4df1-b147-28a52a7d0e62\") " pod="openshift-ingress/e2e-inf-b-506964-openshift-default-5bfdbc5849-4qwwq"
Oct 30 22:31:47.574139 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:47.572786    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64a0ba20-2130-4df1-b147-28a52a7d0e62-workload-socket\") pod \"e2e-inf-b-506964-openshift-default-5bfdbc5849-4qwwq\" (UID: \"64a0ba20-2130-4df1-b147-28a52a7d0e62\") " pod="openshift-ingress/e2e-inf-b-506964-openshift-default-5bfdbc5849-4qwwq"
Oct 30 22:31:47.574139 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:47.572810    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64a0ba20-2130-4df1-b147-28a52a7d0e62-workload-certs\") pod \"e2e-inf-b-506964-openshift-default-5bfdbc5849-4qwwq\" (UID: \"64a0ba20-2130-4df1-b147-28a52a7d0e62\") " pod="openshift-ingress/e2e-inf-b-506964-openshift-default-5bfdbc5849-4qwwq"
Oct 30 22:31:47.574139 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:47.572861    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktbzp\" (UniqueName: \"kubernetes.io/projected/64a0ba20-2130-4df1-b147-28a52a7d0e62-kube-api-access-ktbzp\") pod \"e2e-inf-b-506964-openshift-default-5bfdbc5849-4qwwq\" (UID: \"64a0ba20-2130-4df1-b147-28a52a7d0e62\") " pod="openshift-ingress/e2e-inf-b-506964-openshift-default-5bfdbc5849-4qwwq"
Oct 30 22:31:47.574139 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:47.572928    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64a0ba20-2130-4df1-b147-28a52a7d0e62-credential-socket\") pod \"e2e-inf-b-506964-openshift-default-5bfdbc5849-4qwwq\" (UID: \"64a0ba20-2130-4df1-b147-28a52a7d0e62\") " pod="openshift-ingress/e2e-inf-b-506964-openshift-default-5bfdbc5849-4qwwq"
Oct 30 22:31:47.574139 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:47.573043    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64a0ba20-2130-4df1-b147-28a52a7d0e62-istio-podinfo\") pod \"e2e-inf-b-506964-openshift-default-5bfdbc5849-4qwwq\" (UID: \"64a0ba20-2130-4df1-b147-28a52a7d0e62\") " pod="openshift-ingress/e2e-inf-b-506964-openshift-default-5bfdbc5849-4qwwq"
Oct 30 22:31:47.574139 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:47.573078    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64a0ba20-2130-4df1-b147-28a52a7d0e62-istio-data\") pod \"e2e-inf-b-506964-openshift-default-5bfdbc5849-4qwwq\" (UID: \"64a0ba20-2130-4df1-b147-28a52a7d0e62\") " pod="openshift-ingress/e2e-inf-b-506964-openshift-default-5bfdbc5849-4qwwq"
Oct 30 22:31:47.574139 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:47.573158    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64a0ba20-2130-4df1-b147-28a52a7d0e62-istiod-ca-cert\") pod \"e2e-inf-b-506964-openshift-default-5bfdbc5849-4qwwq\" (UID: \"64a0ba20-2130-4df1-b147-28a52a7d0e62\") " pod="openshift-ingress/e2e-inf-b-506964-openshift-default-5bfdbc5849-4qwwq"
Oct 30 22:31:47.674948 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:47.674842    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64a0ba20-2130-4df1-b147-28a52a7d0e62-credential-socket\") pod \"e2e-inf-b-506964-openshift-default-5bfdbc5849-4qwwq\" (UID: \"64a0ba20-2130-4df1-b147-28a52a7d0e62\") " pod="openshift-ingress/e2e-inf-b-506964-openshift-default-5bfdbc5849-4qwwq"
Oct 30 22:31:47.675135 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:47.675010    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64a0ba20-2130-4df1-b147-28a52a7d0e62-istio-podinfo\") pod \"e2e-inf-b-506964-openshift-default-5bfdbc5849-4qwwq\" (UID: \"64a0ba20-2130-4df1-b147-28a52a7d0e62\") " pod="openshift-ingress/e2e-inf-b-506964-openshift-default-5bfdbc5849-4qwwq"
Oct 30 22:31:47.675135 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:47.675051    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64a0ba20-2130-4df1-b147-28a52a7d0e62-istio-data\") pod \"e2e-inf-b-506964-openshift-default-5bfdbc5849-4qwwq\" (UID: \"64a0ba20-2130-4df1-b147-28a52a7d0e62\") " pod="openshift-ingress/e2e-inf-b-506964-openshift-default-5bfdbc5849-4qwwq"
Oct 30 22:31:47.675283 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:47.675138    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64a0ba20-2130-4df1-b147-28a52a7d0e62-istiod-ca-cert\") pod \"e2e-inf-b-506964-openshift-default-5bfdbc5849-4qwwq\" (UID: \"64a0ba20-2130-4df1-b147-28a52a7d0e62\") " pod="openshift-ingress/e2e-inf-b-506964-openshift-default-5bfdbc5849-4qwwq"
Oct 30 22:31:47.675283 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:47.675193    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64a0ba20-2130-4df1-b147-28a52a7d0e62-istio-token\") pod \"e2e-inf-b-506964-openshift-default-5bfdbc5849-4qwwq\" (UID: \"64a0ba20-2130-4df1-b147-28a52a7d0e62\") " pod="openshift-ingress/e2e-inf-b-506964-openshift-default-5bfdbc5849-4qwwq"
Oct 30 22:31:47.675283 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:47.675249    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64a0ba20-2130-4df1-b147-28a52a7d0e62-istio-envoy\") pod \"e2e-inf-b-506964-openshift-default-5bfdbc5849-4qwwq\" (UID: \"64a0ba20-2130-4df1-b147-28a52a7d0e62\") " pod="openshift-ingress/e2e-inf-b-506964-openshift-default-5bfdbc5849-4qwwq"
Oct 30 22:31:47.675479 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:47.675309    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64a0ba20-2130-4df1-b147-28a52a7d0e62-workload-socket\") pod \"e2e-inf-b-506964-openshift-default-5bfdbc5849-4qwwq\" (UID: \"64a0ba20-2130-4df1-b147-28a52a7d0e62\") " pod="openshift-ingress/e2e-inf-b-506964-openshift-default-5bfdbc5849-4qwwq"
Oct 30 22:31:47.675479 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:47.675345    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64a0ba20-2130-4df1-b147-28a52a7d0e62-workload-certs\") pod \"e2e-inf-b-506964-openshift-default-5bfdbc5849-4qwwq\" (UID: \"64a0ba20-2130-4df1-b147-28a52a7d0e62\") " pod="openshift-ingress/e2e-inf-b-506964-openshift-default-5bfdbc5849-4qwwq"
Oct 30 22:31:47.675479 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:47.675425    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktbzp\" (UniqueName: \"kubernetes.io/projected/64a0ba20-2130-4df1-b147-28a52a7d0e62-kube-api-access-ktbzp\") pod \"e2e-inf-b-506964-openshift-default-5bfdbc5849-4qwwq\" (UID: \"64a0ba20-2130-4df1-b147-28a52a7d0e62\") " pod="openshift-ingress/e2e-inf-b-506964-openshift-default-5bfdbc5849-4qwwq"
Oct 30 22:31:47.676016 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:47.675310    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64a0ba20-2130-4df1-b147-28a52a7d0e62-credential-socket\") pod \"e2e-inf-b-506964-openshift-default-5bfdbc5849-4qwwq\" (UID: \"64a0ba20-2130-4df1-b147-28a52a7d0e62\") " pod="openshift-ingress/e2e-inf-b-506964-openshift-default-5bfdbc5849-4qwwq"
Oct 30 22:31:47.676155 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:47.676070    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64a0ba20-2130-4df1-b147-28a52a7d0e62-istiod-ca-cert\") pod \"e2e-inf-b-506964-openshift-default-5bfdbc5849-4qwwq\" (UID: \"64a0ba20-2130-4df1-b147-28a52a7d0e62\") " pod="openshift-ingress/e2e-inf-b-506964-openshift-default-5bfdbc5849-4qwwq"
Oct 30 22:31:47.676371 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:47.676347    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64a0ba20-2130-4df1-b147-28a52a7d0e62-workload-socket\") pod \"e2e-inf-b-506964-openshift-default-5bfdbc5849-4qwwq\" (UID: \"64a0ba20-2130-4df1-b147-28a52a7d0e62\") " pod="openshift-ingress/e2e-inf-b-506964-openshift-default-5bfdbc5849-4qwwq"
Oct 30 22:31:47.676619 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:47.676595    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64a0ba20-2130-4df1-b147-28a52a7d0e62-istio-data\") pod \"e2e-inf-b-506964-openshift-default-5bfdbc5849-4qwwq\" (UID: \"64a0ba20-2130-4df1-b147-28a52a7d0e62\") " pod="openshift-ingress/e2e-inf-b-506964-openshift-default-5bfdbc5849-4qwwq"
Oct 30 22:31:47.677004 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:47.676979    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64a0ba20-2130-4df1-b147-28a52a7d0e62-workload-certs\") pod \"e2e-inf-b-506964-openshift-default-5bfdbc5849-4qwwq\" (UID: \"64a0ba20-2130-4df1-b147-28a52a7d0e62\") " pod="openshift-ingress/e2e-inf-b-506964-openshift-default-5bfdbc5849-4qwwq"
Oct 30 22:31:47.678251 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:47.678230    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64a0ba20-2130-4df1-b147-28a52a7d0e62-istio-envoy\") pod \"e2e-inf-b-506964-openshift-default-5bfdbc5849-4qwwq\" (UID: \"64a0ba20-2130-4df1-b147-28a52a7d0e62\") " pod="openshift-ingress/e2e-inf-b-506964-openshift-default-5bfdbc5849-4qwwq"
Oct 30 22:31:47.679044 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:47.679018    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64a0ba20-2130-4df1-b147-28a52a7d0e62-istio-podinfo\") pod \"e2e-inf-b-506964-openshift-default-5bfdbc5849-4qwwq\" (UID: \"64a0ba20-2130-4df1-b147-28a52a7d0e62\") " pod="openshift-ingress/e2e-inf-b-506964-openshift-default-5bfdbc5849-4qwwq"
Oct 30 22:31:47.736840 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:47.736813    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64a0ba20-2130-4df1-b147-28a52a7d0e62-istio-token\") pod \"e2e-inf-b-506964-openshift-default-5bfdbc5849-4qwwq\" (UID: \"64a0ba20-2130-4df1-b147-28a52a7d0e62\") " pod="openshift-ingress/e2e-inf-b-506964-openshift-default-5bfdbc5849-4qwwq"
Oct 30 22:31:47.736989 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:47.736884    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktbzp\" (UniqueName: \"kubernetes.io/projected/64a0ba20-2130-4df1-b147-28a52a7d0e62-kube-api-access-ktbzp\") pod \"e2e-inf-b-506964-openshift-default-5bfdbc5849-4qwwq\" (UID: \"64a0ba20-2130-4df1-b147-28a52a7d0e62\") " pod="openshift-ingress/e2e-inf-b-506964-openshift-default-5bfdbc5849-4qwwq"
Oct 30 22:31:47.750472 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:47.750196    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-506964-openshift-default-5bfdbc5849-4qwwq"
Oct 30 22:31:48.224701 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:48.224669    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-506964-openshift-default-5bfdbc5849-4qwwq"]
Oct 30 22:31:48.230295 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:31:48.230268    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64a0ba20_2130_4df1_b147_28a52a7d0e62.slice/crio-5fa835ed7ad27e0b73b0fdd0e7d3886130505dd80175fdde672583c9e89563b2 WatchSource:0}: Error finding container 5fa835ed7ad27e0b73b0fdd0e7d3886130505dd80175fdde672583c9e89563b2: Status 404 returned error can't find the container with id 5fa835ed7ad27e0b73b0fdd0e7d3886130505dd80175fdde672583c9e89563b2
Oct 30 22:31:48.232821 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:48.232789    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:31:48.232897 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:48.232852    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:31:48.232939 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:48.232894    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:31:48.280817 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:48.280789    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-506964-openshift-default-5bfdbc5849-4qwwq" event={"ID":"64a0ba20-2130-4df1-b147-28a52a7d0e62","Type":"ContainerStarted","Data":"5fa835ed7ad27e0b73b0fdd0e7d3886130505dd80175fdde672583c9e89563b2"}
Oct 30 22:31:49.289124 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:49.289084    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-506964-openshift-default-5bfdbc5849-4qwwq" event={"ID":"64a0ba20-2130-4df1-b147-28a52a7d0e62","Type":"ContainerStarted","Data":"03a0781f38eb9c11adf9a9b59c0598959af5afd188736fd885086d08384bc326"}
Oct 30 22:31:49.506244 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:49.506188    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-506964-openshift-default-5bfdbc5849-4qwwq" podStartSLOduration=2.506172656 podStartE2EDuration="2.506172656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:31:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:31:49.456193664 +0000 UTC m=+2487.060501216" watchObservedRunningTime="2026-10-30 22:31:49.506172656 +0000 UTC m=+2487.110480209"
Oct 30 22:31:49.751526 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:49.751495    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-506964-openshift-default-5bfdbc5849-4qwwq"
Oct 30 22:31:49.757458 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:49.757432    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-506964-openshift-default-5bfdbc5849-4qwwq"
Oct 30 22:31:50.296344 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:50.296301    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-506964-openshift-default-5bfdbc5849-4qwwq"
Oct 30 22:31:50.297210 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:50.297192    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-506964-openshift-default-5bfdbc5849-4qwwq"
Oct 30 22:31:51.640344 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:51.640311    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-5686cb-openshift-default-5f6d8dbdbc-s8jnc"]
Oct 30 22:31:51.640767 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:51.640550    5678 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-5686cb-openshift-default-5f6d8dbdbc-s8jnc" podUID="707b0553-ce35-4de0-ba4a-fffa4adfc6ee" containerName="istio-proxy" containerID="cri-o://84dba6a3fd54fd611ac8afeb27d450cd35128264db4e2356b46dee14fec5a6d0" gracePeriod=30
Oct 30 22:31:55.844097 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:55.844065    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-506964-7dc79f55ff-wm696"]
Oct 30 22:31:55.848332 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:55.848312    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-506964-7dc79f55ff-wm696"
Oct 30 22:31:55.900503 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:55.900461    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-506964\""
Oct 30 22:31:55.911563 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:55.911375    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1e95a6d-29c6-4c95-b537-1037089dcab8-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-506964-7dc79f55ff-wm696\" (UID: \"f1e95a6d-29c6-4c95-b537-1037089dcab8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-506964-7dc79f55ff-wm696"
Oct 30 22:31:55.911936 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:55.911915    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8p2tw\" (UniqueName: \"kubernetes.io/projected/f1e95a6d-29c6-4c95-b537-1037089dcab8-kube-api-access-8p2tw\") pod \"payload-pre-processing-e2e-inf-b-506964-7dc79f55ff-wm696\" (UID: \"f1e95a6d-29c6-4c95-b537-1037089dcab8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-506964-7dc79f55ff-wm696"
Oct 30 22:31:55.951577 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:55.951531    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-506964-7dc79f55ff-wm696"]
Oct 30 22:31:55.951782 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:55.951711    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-506964-dockercfg-6gjjr\""
Oct 30 22:31:56.013904 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:56.013872    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8p2tw\" (UniqueName: \"kubernetes.io/projected/f1e95a6d-29c6-4c95-b537-1037089dcab8-kube-api-access-8p2tw\") pod \"payload-pre-processing-e2e-inf-b-506964-7dc79f55ff-wm696\" (UID: \"f1e95a6d-29c6-4c95-b537-1037089dcab8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-506964-7dc79f55ff-wm696"
Oct 30 22:31:56.014322 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:56.014298    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1e95a6d-29c6-4c95-b537-1037089dcab8-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-506964-7dc79f55ff-wm696\" (UID: \"f1e95a6d-29c6-4c95-b537-1037089dcab8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-506964-7dc79f55ff-wm696"
Oct 30 22:31:56.015237 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:56.015213    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1e95a6d-29c6-4c95-b537-1037089dcab8-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-506964-7dc79f55ff-wm696\" (UID: \"f1e95a6d-29c6-4c95-b537-1037089dcab8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-506964-7dc79f55ff-wm696"
Oct 30 22:31:56.072591 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:56.072533    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8p2tw\" (UniqueName: \"kubernetes.io/projected/f1e95a6d-29c6-4c95-b537-1037089dcab8-kube-api-access-8p2tw\") pod \"payload-pre-processing-e2e-inf-b-506964-7dc79f55ff-wm696\" (UID: \"f1e95a6d-29c6-4c95-b537-1037089dcab8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-506964-7dc79f55ff-wm696"
Oct 30 22:31:56.163582 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:56.163552    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-506964-7dc79f55ff-wm696"
Oct 30 22:31:56.550682 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:56.550598    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-506964-7dc79f55ff-wm696"]
Oct 30 22:31:56.557013 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:31:56.556982    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1e95a6d_29c6_4c95_b537_1037089dcab8.slice/crio-98df0c01beaa13055720b5b6a2561d9699bc5bbfdb20c92827e82694b31c55ff WatchSource:0}: Error finding container 98df0c01beaa13055720b5b6a2561d9699bc5bbfdb20c92827e82694b31c55ff: Status 404 returned error can't find the container with id 98df0c01beaa13055720b5b6a2561d9699bc5bbfdb20c92827e82694b31c55ff
Oct 30 22:31:56.635032 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:56.635006    5678 patch_prober.go:29] interesting pod/e2e-api-b-5686cb-openshift-default-5f6d8dbdbc-s8jnc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:31:56.635143 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:56.635067    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-5686cb-openshift-default-5f6d8dbdbc-s8jnc" podUID="707b0553-ce35-4de0-ba4a-fffa4adfc6ee" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:31:56.894411 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:56.894375    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-5686cb-openshift-default-5f6d8dbdbc-s8jnc"
Oct 30 22:31:57.028590 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:57.028560    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/707b0553-ce35-4de0-ba4a-fffa4adfc6ee-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/707b0553-ce35-4de0-ba4a-fffa4adfc6ee-istio-envoy\") pod \"707b0553-ce35-4de0-ba4a-fffa4adfc6ee\" (UID: \"707b0553-ce35-4de0-ba4a-fffa4adfc6ee\") "
Oct 30 22:31:57.028772 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:57.028630    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/707b0553-ce35-4de0-ba4a-fffa4adfc6ee-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/707b0553-ce35-4de0-ba4a-fffa4adfc6ee-credential-socket\") pod \"707b0553-ce35-4de0-ba4a-fffa4adfc6ee\" (UID: \"707b0553-ce35-4de0-ba4a-fffa4adfc6ee\") "
Oct 30 22:31:57.028772 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:57.028681    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/707b0553-ce35-4de0-ba4a-fffa4adfc6ee-kube-api-access-w2xz5\" (UniqueName: \"kubernetes.io/projected/707b0553-ce35-4de0-ba4a-fffa4adfc6ee-kube-api-access-w2xz5\") pod \"707b0553-ce35-4de0-ba4a-fffa4adfc6ee\" (UID: \"707b0553-ce35-4de0-ba4a-fffa4adfc6ee\") "
Oct 30 22:31:57.028925 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:57.028809    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/707b0553-ce35-4de0-ba4a-fffa4adfc6ee-istio-token\" (UniqueName: \"kubernetes.io/projected/707b0553-ce35-4de0-ba4a-fffa4adfc6ee-istio-token\") pod \"707b0553-ce35-4de0-ba4a-fffa4adfc6ee\" (UID: \"707b0553-ce35-4de0-ba4a-fffa4adfc6ee\") "
Oct 30 22:31:57.028925 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:57.028880    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/707b0553-ce35-4de0-ba4a-fffa4adfc6ee-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/707b0553-ce35-4de0-ba4a-fffa4adfc6ee-istio-data\") pod \"707b0553-ce35-4de0-ba4a-fffa4adfc6ee\" (UID: \"707b0553-ce35-4de0-ba4a-fffa4adfc6ee\") "
Oct 30 22:31:57.028925 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:57.028918    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/707b0553-ce35-4de0-ba4a-fffa4adfc6ee-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/707b0553-ce35-4de0-ba4a-fffa4adfc6ee-workload-certs\") pod \"707b0553-ce35-4de0-ba4a-fffa4adfc6ee\" (UID: \"707b0553-ce35-4de0-ba4a-fffa4adfc6ee\") "
Oct 30 22:31:57.029100 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:57.028917    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/707b0553-ce35-4de0-ba4a-fffa4adfc6ee-credential-socket" pod "707b0553-ce35-4de0-ba4a-fffa4adfc6ee" (UID: "707b0553-ce35-4de0-ba4a-fffa4adfc6ee"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:57.029100 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:57.029012    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/707b0553-ce35-4de0-ba4a-fffa4adfc6ee-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/707b0553-ce35-4de0-ba4a-fffa4adfc6ee-istio-podinfo\") pod \"707b0553-ce35-4de0-ba4a-fffa4adfc6ee\" (UID: \"707b0553-ce35-4de0-ba4a-fffa4adfc6ee\") "
Oct 30 22:31:57.029218 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:57.029110    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/707b0553-ce35-4de0-ba4a-fffa4adfc6ee-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/707b0553-ce35-4de0-ba4a-fffa4adfc6ee-istiod-ca-cert\") pod \"707b0553-ce35-4de0-ba4a-fffa4adfc6ee\" (UID: \"707b0553-ce35-4de0-ba4a-fffa4adfc6ee\") "
Oct 30 22:31:57.029218 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:57.029134    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/707b0553-ce35-4de0-ba4a-fffa4adfc6ee-workload-certs" pod "707b0553-ce35-4de0-ba4a-fffa4adfc6ee" (UID: "707b0553-ce35-4de0-ba4a-fffa4adfc6ee"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:57.029218 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:57.029145    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/707b0553-ce35-4de0-ba4a-fffa4adfc6ee-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/707b0553-ce35-4de0-ba4a-fffa4adfc6ee-workload-socket\") pod \"707b0553-ce35-4de0-ba4a-fffa4adfc6ee\" (UID: \"707b0553-ce35-4de0-ba4a-fffa4adfc6ee\") "
Oct 30 22:31:57.029394 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:57.029323    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/707b0553-ce35-4de0-ba4a-fffa4adfc6ee-istio-data" pod "707b0553-ce35-4de0-ba4a-fffa4adfc6ee" (UID: "707b0553-ce35-4de0-ba4a-fffa4adfc6ee"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:57.029608 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:57.029578    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/707b0553-ce35-4de0-ba4a-fffa4adfc6ee-workload-socket" pod "707b0553-ce35-4de0-ba4a-fffa4adfc6ee" (UID: "707b0553-ce35-4de0-ba4a-fffa4adfc6ee"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:57.029703 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:57.029624    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/707b0553-ce35-4de0-ba4a-fffa4adfc6ee-istiod-ca-cert" pod "707b0553-ce35-4de0-ba4a-fffa4adfc6ee" (UID: "707b0553-ce35-4de0-ba4a-fffa4adfc6ee"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:31:57.029852 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:57.029830    5678 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/707b0553-ce35-4de0-ba4a-fffa4adfc6ee-istio-data\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:57.029852 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:57.029854    5678 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/707b0553-ce35-4de0-ba4a-fffa4adfc6ee-workload-certs\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:57.030031 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:57.029867    5678 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/707b0553-ce35-4de0-ba4a-fffa4adfc6ee-istiod-ca-cert\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:57.030031 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:57.029882    5678 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/707b0553-ce35-4de0-ba4a-fffa4adfc6ee-workload-socket\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:57.030031 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:57.029896    5678 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/707b0553-ce35-4de0-ba4a-fffa4adfc6ee-credential-socket\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:57.031048 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:57.031019    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/707b0553-ce35-4de0-ba4a-fffa4adfc6ee-istio-envoy" pod "707b0553-ce35-4de0-ba4a-fffa4adfc6ee" (UID: "707b0553-ce35-4de0-ba4a-fffa4adfc6ee"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:57.031842 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:57.031815    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/707b0553-ce35-4de0-ba4a-fffa4adfc6ee-istio-token" pod "707b0553-ce35-4de0-ba4a-fffa4adfc6ee" (UID: "707b0553-ce35-4de0-ba4a-fffa4adfc6ee"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:31:57.031986 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:57.031965    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/707b0553-ce35-4de0-ba4a-fffa4adfc6ee-kube-api-access-w2xz5" pod "707b0553-ce35-4de0-ba4a-fffa4adfc6ee" (UID: "707b0553-ce35-4de0-ba4a-fffa4adfc6ee"). InnerVolumeSpecName "kube-api-access-w2xz5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:31:57.032063 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:57.032027    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/707b0553-ce35-4de0-ba4a-fffa4adfc6ee-istio-podinfo" pod "707b0553-ce35-4de0-ba4a-fffa4adfc6ee" (UID: "707b0553-ce35-4de0-ba4a-fffa4adfc6ee"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:31:57.131313 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:57.131269    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2xz5\" (UniqueName: \"kubernetes.io/projected/707b0553-ce35-4de0-ba4a-fffa4adfc6ee-kube-api-access-w2xz5\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:57.131313 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:57.131311    5678 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/707b0553-ce35-4de0-ba4a-fffa4adfc6ee-istio-token\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:57.131313 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:57.131321    5678 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/707b0553-ce35-4de0-ba4a-fffa4adfc6ee-istio-podinfo\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:57.131583 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:57.131332    5678 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/707b0553-ce35-4de0-ba4a-fffa4adfc6ee-istio-envoy\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:57.342454 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:57.342341    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-506964-7dc79f55ff-wm696" event={"ID":"f1e95a6d-29c6-4c95-b537-1037089dcab8","Type":"ContainerStarted","Data":"11e5d17edc65c348579a6d9a79634fdd742105d6cc8c7f8c7c3bcc9921397e67"}
Oct 30 22:31:57.342622 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:57.342460    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-506964-7dc79f55ff-wm696"
Oct 30 22:31:57.342622 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:57.342479    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-506964-7dc79f55ff-wm696" event={"ID":"f1e95a6d-29c6-4c95-b537-1037089dcab8","Type":"ContainerStarted","Data":"98df0c01beaa13055720b5b6a2561d9699bc5bbfdb20c92827e82694b31c55ff"}
Oct 30 22:31:57.344199 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:57.344170    5678 generic.go:356] "Generic (PLEG): container finished" podID="707b0553-ce35-4de0-ba4a-fffa4adfc6ee" containerID="84dba6a3fd54fd611ac8afeb27d450cd35128264db4e2356b46dee14fec5a6d0" exitCode=0
Oct 30 22:31:57.344318 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:57.344213    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-5686cb-openshift-default-5f6d8dbdbc-s8jnc"
Oct 30 22:31:57.344318 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:57.344239    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-5686cb-openshift-default-5f6d8dbdbc-s8jnc" event={"ID":"707b0553-ce35-4de0-ba4a-fffa4adfc6ee","Type":"ContainerDied","Data":"84dba6a3fd54fd611ac8afeb27d450cd35128264db4e2356b46dee14fec5a6d0"}
Oct 30 22:31:57.344318 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:57.344282    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-5686cb-openshift-default-5f6d8dbdbc-s8jnc" event={"ID":"707b0553-ce35-4de0-ba4a-fffa4adfc6ee","Type":"ContainerDied","Data":"4288eaafb1597b99650ed433f410afa0fc814d0e3a04f987be37915cb1a1abde"}
Oct 30 22:31:57.344318 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:57.344304    5678 scope.go:122] "RemoveContainer" containerID="84dba6a3fd54fd611ac8afeb27d450cd35128264db4e2356b46dee14fec5a6d0"
Oct 30 22:31:57.355745 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:57.355729    5678 scope.go:122] "RemoveContainer" containerID="1dc97660aff8ac9223122df34c70ae1232b4e4ba5e5378e1b72098e2e8e770ec"
Oct 30 22:31:57.365784 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:57.365767    5678 scope.go:122] "RemoveContainer" containerID="84dba6a3fd54fd611ac8afeb27d450cd35128264db4e2356b46dee14fec5a6d0"
Oct 30 22:31:57.366066 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:31:57.366043    5678 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84dba6a3fd54fd611ac8afeb27d450cd35128264db4e2356b46dee14fec5a6d0\": container with ID starting with 84dba6a3fd54fd611ac8afeb27d450cd35128264db4e2356b46dee14fec5a6d0 not found: ID does not exist" containerID="84dba6a3fd54fd611ac8afeb27d450cd35128264db4e2356b46dee14fec5a6d0"
Oct 30 22:31:57.366177 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:57.366072    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84dba6a3fd54fd611ac8afeb27d450cd35128264db4e2356b46dee14fec5a6d0"} err="failed to get container status \"84dba6a3fd54fd611ac8afeb27d450cd35128264db4e2356b46dee14fec5a6d0\": rpc error: code = NotFound desc = could not find container \"84dba6a3fd54fd611ac8afeb27d450cd35128264db4e2356b46dee14fec5a6d0\": container with ID starting with 84dba6a3fd54fd611ac8afeb27d450cd35128264db4e2356b46dee14fec5a6d0 not found: ID does not exist"
Oct 30 22:31:57.366177 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:57.366089    5678 scope.go:122] "RemoveContainer" containerID="1dc97660aff8ac9223122df34c70ae1232b4e4ba5e5378e1b72098e2e8e770ec"
Oct 30 22:31:57.366364 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:31:57.366350    5678 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1dc97660aff8ac9223122df34c70ae1232b4e4ba5e5378e1b72098e2e8e770ec\": container with ID starting with 1dc97660aff8ac9223122df34c70ae1232b4e4ba5e5378e1b72098e2e8e770ec not found: ID does not exist" containerID="1dc97660aff8ac9223122df34c70ae1232b4e4ba5e5378e1b72098e2e8e770ec"
Oct 30 22:31:57.366426 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:57.366367    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1dc97660aff8ac9223122df34c70ae1232b4e4ba5e5378e1b72098e2e8e770ec"} err="failed to get container status \"1dc97660aff8ac9223122df34c70ae1232b4e4ba5e5378e1b72098e2e8e770ec\": rpc error: code = NotFound desc = could not find container \"1dc97660aff8ac9223122df34c70ae1232b4e4ba5e5378e1b72098e2e8e770ec\": container with ID starting with 1dc97660aff8ac9223122df34c70ae1232b4e4ba5e5378e1b72098e2e8e770ec not found: ID does not exist"
Oct 30 22:31:57.456699 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:57.456645    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-506964-7dc79f55ff-wm696" podStartSLOduration=2.456632811 podStartE2EDuration="2.456632811s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:31:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:31:57.453708564 +0000 UTC m=+2495.058016118" watchObservedRunningTime="2026-10-30 22:31:57.456632811 +0000 UTC m=+2495.060940394"
Oct 30 22:31:57.563320 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:57.563288    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-5686cb-openshift-default-5f6d8dbdbc-s8jnc"]
Oct 30 22:31:57.610635 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:57.610552    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-5686cb-openshift-default-5f6d8dbdbc-s8jnc"]
Oct 30 22:31:58.522423 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:31:58.522374    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="707b0553-ce35-4de0-ba4a-fffa4adfc6ee" path="/var/lib/kubelet/pods/707b0553-ce35-4de0-ba4a-fffa4adfc6ee/volumes"
Oct 30 22:32:03.274884 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:03.274847    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-c42c0a-55d8fcbd6c-rg6mj"]
Oct 30 22:32:03.375644 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:03.375604    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-c42c0a-5694bffdff-4gb6s"]
Oct 30 22:32:08.275163 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:08.275122    5678 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-c42c0a-55d8fcbd6c-rg6mj" podUID="bf53af2e-175e-4af2-a5a4-144853f6af11" containerName="payload-processing" containerID="cri-o://d3e4c658ab0439c22a104825bb226c4cf031bae0873256f67d10a4b84f6cf30b" gracePeriod=25
Oct 30 22:32:08.355682 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:08.355656    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-506964-7dc79f55ff-wm696"
Oct 30 22:32:08.376517 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:08.376482    5678 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-c42c0a-5694bffdff-4gb6s" podUID="195e9bc1-18c1-4fcc-9999-2518f332c8c5" containerName="payload-pre-processing" containerID="cri-o://464bce49761620adf12fd89d4065c6d620555fc5ca4f81db9249bc1aaef59116" gracePeriod=25
Oct 30 22:32:08.420896 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:08.420864    5678 generic.go:356] "Generic (PLEG): container finished" podID="bf53af2e-175e-4af2-a5a4-144853f6af11" containerID="d3e4c658ab0439c22a104825bb226c4cf031bae0873256f67d10a4b84f6cf30b" exitCode=0
Oct 30 22:32:08.421053 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:08.420939    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-c42c0a-55d8fcbd6c-rg6mj" event={"ID":"bf53af2e-175e-4af2-a5a4-144853f6af11","Type":"ContainerDied","Data":"d3e4c658ab0439c22a104825bb226c4cf031bae0873256f67d10a4b84f6cf30b"}
Oct 30 22:32:08.553261 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:08.553234    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-c42c0a-55d8fcbd6c-rg6mj"
Oct 30 22:32:08.569016 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:08.568985    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf53af2e-175e-4af2-a5a4-144853f6af11-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf53af2e-175e-4af2-a5a4-144853f6af11-plugins-config-volume\") pod \"bf53af2e-175e-4af2-a5a4-144853f6af11\" (UID: \"bf53af2e-175e-4af2-a5a4-144853f6af11\") "
Oct 30 22:32:08.569448 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:08.569334    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf53af2e-175e-4af2-a5a4-144853f6af11-plugins-config-volume" pod "bf53af2e-175e-4af2-a5a4-144853f6af11" (UID: "bf53af2e-175e-4af2-a5a4-144853f6af11"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:32:08.569567 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:08.569541    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf53af2e-175e-4af2-a5a4-144853f6af11-kube-api-access-8zm5b\" (UniqueName: \"kubernetes.io/projected/bf53af2e-175e-4af2-a5a4-144853f6af11-kube-api-access-8zm5b\") pod \"bf53af2e-175e-4af2-a5a4-144853f6af11\" (UID: \"bf53af2e-175e-4af2-a5a4-144853f6af11\") "
Oct 30 22:32:08.569982 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:08.569959    5678 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf53af2e-175e-4af2-a5a4-144853f6af11-plugins-config-volume\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:32:08.572658 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:08.572592    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf53af2e-175e-4af2-a5a4-144853f6af11-kube-api-access-8zm5b" pod "bf53af2e-175e-4af2-a5a4-144853f6af11" (UID: "bf53af2e-175e-4af2-a5a4-144853f6af11"). InnerVolumeSpecName "kube-api-access-8zm5b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:32:08.638993 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:08.638966    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-c42c0a-5694bffdff-4gb6s"
Oct 30 22:32:08.670585 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:08.670554    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/195e9bc1-18c1-4fcc-9999-2518f332c8c5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/195e9bc1-18c1-4fcc-9999-2518f332c8c5-plugins-config-volume\") pod \"195e9bc1-18c1-4fcc-9999-2518f332c8c5\" (UID: \"195e9bc1-18c1-4fcc-9999-2518f332c8c5\") "
Oct 30 22:32:08.670801 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:08.670636    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/195e9bc1-18c1-4fcc-9999-2518f332c8c5-kube-api-access-m7bwx\" (UniqueName: \"kubernetes.io/projected/195e9bc1-18c1-4fcc-9999-2518f332c8c5-kube-api-access-m7bwx\") pod \"195e9bc1-18c1-4fcc-9999-2518f332c8c5\" (UID: \"195e9bc1-18c1-4fcc-9999-2518f332c8c5\") "
Oct 30 22:32:08.670999 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:08.670974    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/195e9bc1-18c1-4fcc-9999-2518f332c8c5-plugins-config-volume" pod "195e9bc1-18c1-4fcc-9999-2518f332c8c5" (UID: "195e9bc1-18c1-4fcc-9999-2518f332c8c5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:32:08.671196 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:08.671177    5678 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/195e9bc1-18c1-4fcc-9999-2518f332c8c5-plugins-config-volume\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:32:08.671273 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:08.671201    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zm5b\" (UniqueName: \"kubernetes.io/projected/bf53af2e-175e-4af2-a5a4-144853f6af11-kube-api-access-8zm5b\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:32:08.672803 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:08.672777    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/195e9bc1-18c1-4fcc-9999-2518f332c8c5-kube-api-access-m7bwx" pod "195e9bc1-18c1-4fcc-9999-2518f332c8c5" (UID: "195e9bc1-18c1-4fcc-9999-2518f332c8c5"). InnerVolumeSpecName "kube-api-access-m7bwx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:32:08.772659 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:08.772624    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7bwx\" (UniqueName: \"kubernetes.io/projected/195e9bc1-18c1-4fcc-9999-2518f332c8c5-kube-api-access-m7bwx\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:32:09.428735 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:09.428700    5678 generic.go:356] "Generic (PLEG): container finished" podID="195e9bc1-18c1-4fcc-9999-2518f332c8c5" containerID="464bce49761620adf12fd89d4065c6d620555fc5ca4f81db9249bc1aaef59116" exitCode=0
Oct 30 22:32:09.429155 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:09.428752    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-c42c0a-5694bffdff-4gb6s"
Oct 30 22:32:09.429155 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:09.428783    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-c42c0a-5694bffdff-4gb6s" event={"ID":"195e9bc1-18c1-4fcc-9999-2518f332c8c5","Type":"ContainerDied","Data":"464bce49761620adf12fd89d4065c6d620555fc5ca4f81db9249bc1aaef59116"}
Oct 30 22:32:09.429155 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:09.428840    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-c42c0a-5694bffdff-4gb6s" event={"ID":"195e9bc1-18c1-4fcc-9999-2518f332c8c5","Type":"ContainerDied","Data":"c64e2e0884721a61260d75d3dfed21c66776059e5ae14b3aa70d19f48be105dd"}
Oct 30 22:32:09.429155 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:09.428879    5678 scope.go:122] "RemoveContainer" containerID="464bce49761620adf12fd89d4065c6d620555fc5ca4f81db9249bc1aaef59116"
Oct 30 22:32:09.430349 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:09.430333    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-c42c0a-55d8fcbd6c-rg6mj"
Oct 30 22:32:09.430464 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:09.430334    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-c42c0a-55d8fcbd6c-rg6mj" event={"ID":"bf53af2e-175e-4af2-a5a4-144853f6af11","Type":"ContainerDied","Data":"b83eec17b7630c730523c71ae20ddce3bdd8dfd36300caefd60100d401725d7f"}
Oct 30 22:32:09.440452 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:09.440431    5678 scope.go:122] "RemoveContainer" containerID="464bce49761620adf12fd89d4065c6d620555fc5ca4f81db9249bc1aaef59116"
Oct 30 22:32:09.440747 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:32:09.440721    5678 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"464bce49761620adf12fd89d4065c6d620555fc5ca4f81db9249bc1aaef59116\": container with ID starting with 464bce49761620adf12fd89d4065c6d620555fc5ca4f81db9249bc1aaef59116 not found: ID does not exist" containerID="464bce49761620adf12fd89d4065c6d620555fc5ca4f81db9249bc1aaef59116"
Oct 30 22:32:09.440838 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:09.440758    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"464bce49761620adf12fd89d4065c6d620555fc5ca4f81db9249bc1aaef59116"} err="failed to get container status \"464bce49761620adf12fd89d4065c6d620555fc5ca4f81db9249bc1aaef59116\": rpc error: code = NotFound desc = could not find container \"464bce49761620adf12fd89d4065c6d620555fc5ca4f81db9249bc1aaef59116\": container with ID starting with 464bce49761620adf12fd89d4065c6d620555fc5ca4f81db9249bc1aaef59116 not found: ID does not exist"
Oct 30 22:32:09.440838 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:09.440779    5678 scope.go:122] "RemoveContainer" containerID="d3e4c658ab0439c22a104825bb226c4cf031bae0873256f67d10a4b84f6cf30b"
Oct 30 22:32:09.588692 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:09.588620    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-c42c0a-5694bffdff-4gb6s"]
Oct 30 22:32:09.636895 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:09.636867    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-c42c0a-5694bffdff-4gb6s"]
Oct 30 22:32:09.700273 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:09.700246    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-c42c0a-55d8fcbd6c-rg6mj"]
Oct 30 22:32:09.705582 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:09.705546    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-c42c0a-55d8fcbd6c-rg6mj"]
Oct 30 22:32:10.524286 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:10.524253    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="195e9bc1-18c1-4fcc-9999-2518f332c8c5" path="/var/lib/kubelet/pods/195e9bc1-18c1-4fcc-9999-2518f332c8c5/volumes"
Oct 30 22:32:10.524766 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:10.524748    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf53af2e-175e-4af2-a5a4-144853f6af11" path="/var/lib/kubelet/pods/bf53af2e-175e-4af2-a5a4-144853f6af11/volumes"
Oct 30 22:32:10.571854 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:32:10.571818    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 22:32:22.668578 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:22.668545    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-07e9c8-f548454dc-lsx4x"]
Oct 30 22:32:22.669079 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:22.669057    5678 cpu_manager.go:414] "containerMap: removing container" podUID="bf53af2e-175e-4af2-a5a4-144853f6af11" containerName="payload-processing"
Oct 30 22:32:22.669079 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:22.669068    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf53af2e-175e-4af2-a5a4-144853f6af11" containerName="payload-processing"
Oct 30 22:32:22.669170 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:22.669082    5678 cpu_manager.go:414] "containerMap: removing container" podUID="195e9bc1-18c1-4fcc-9999-2518f332c8c5" containerName="payload-pre-processing"
Oct 30 22:32:22.669170 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:22.669088    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="195e9bc1-18c1-4fcc-9999-2518f332c8c5" containerName="payload-pre-processing"
Oct 30 22:32:22.669170 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:22.669116    5678 cpu_manager.go:414] "containerMap: removing container" podUID="707b0553-ce35-4de0-ba4a-fffa4adfc6ee" containerName="istio-proxy"
Oct 30 22:32:22.669170 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:22.669121    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="707b0553-ce35-4de0-ba4a-fffa4adfc6ee" containerName="istio-proxy"
Oct 30 22:32:22.669170 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:22.669134    5678 cpu_manager.go:414] "containerMap: removing container" podUID="707b0553-ce35-4de0-ba4a-fffa4adfc6ee" containerName="istio-proxy"
Oct 30 22:32:22.669170 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:22.669139    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="707b0553-ce35-4de0-ba4a-fffa4adfc6ee" containerName="istio-proxy"
Oct 30 22:32:22.669360 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:22.669213    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf53af2e-175e-4af2-a5a4-144853f6af11" containerName="payload-processing"
Oct 30 22:32:22.669360 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:22.669221    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="195e9bc1-18c1-4fcc-9999-2518f332c8c5" containerName="payload-pre-processing"
Oct 30 22:32:22.669360 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:22.669233    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="707b0553-ce35-4de0-ba4a-fffa4adfc6ee" containerName="istio-proxy"
Oct 30 22:32:22.672788 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:22.672767    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-07e9c8-f548454dc-lsx4x"
Oct 30 22:32:22.723565 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:22.723543    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-07e9c8-serving-cert\""
Oct 30 22:32:22.730553 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:22.730529    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-07e9c8-f548454dc-lsx4x"]
Oct 30 22:32:22.741161 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:22.741133    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rvs8\" (UniqueName: \"kubernetes.io/projected/c01c1a4e-7d4d-45ca-b92e-c1cdd9013059-kube-api-access-5rvs8\") pod \"maas-api-e2e-inf-b-07e9c8-f548454dc-lsx4x\" (UID: \"c01c1a4e-7d4d-45ca-b92e-c1cdd9013059\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-07e9c8-f548454dc-lsx4x"
Oct 30 22:32:22.741294 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:22.741167    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c01c1a4e-7d4d-45ca-b92e-c1cdd9013059-metrics-certs\") pod \"maas-api-e2e-inf-b-07e9c8-f548454dc-lsx4x\" (UID: \"c01c1a4e-7d4d-45ca-b92e-c1cdd9013059\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-07e9c8-f548454dc-lsx4x"
Oct 30 22:32:22.741377 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:22.741354    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c01c1a4e-7d4d-45ca-b92e-c1cdd9013059-maas-api-tls\") pod \"maas-api-e2e-inf-b-07e9c8-f548454dc-lsx4x\" (UID: \"c01c1a4e-7d4d-45ca-b92e-c1cdd9013059\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-07e9c8-f548454dc-lsx4x"
Oct 30 22:32:22.741466 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:22.741438    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c01c1a4e-7d4d-45ca-b92e-c1cdd9013059-openshift-service-ca\") pod \"maas-api-e2e-inf-b-07e9c8-f548454dc-lsx4x\" (UID: \"c01c1a4e-7d4d-45ca-b92e-c1cdd9013059\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-07e9c8-f548454dc-lsx4x"
Oct 30 22:32:22.842413 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:22.842370    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rvs8\" (UniqueName: \"kubernetes.io/projected/c01c1a4e-7d4d-45ca-b92e-c1cdd9013059-kube-api-access-5rvs8\") pod \"maas-api-e2e-inf-b-07e9c8-f548454dc-lsx4x\" (UID: \"c01c1a4e-7d4d-45ca-b92e-c1cdd9013059\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-07e9c8-f548454dc-lsx4x"
Oct 30 22:32:22.842587 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:22.842450    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c01c1a4e-7d4d-45ca-b92e-c1cdd9013059-metrics-certs\") pod \"maas-api-e2e-inf-b-07e9c8-f548454dc-lsx4x\" (UID: \"c01c1a4e-7d4d-45ca-b92e-c1cdd9013059\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-07e9c8-f548454dc-lsx4x"
Oct 30 22:32:22.842656 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:22.842642    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c01c1a4e-7d4d-45ca-b92e-c1cdd9013059-maas-api-tls\") pod \"maas-api-e2e-inf-b-07e9c8-f548454dc-lsx4x\" (UID: \"c01c1a4e-7d4d-45ca-b92e-c1cdd9013059\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-07e9c8-f548454dc-lsx4x"
Oct 30 22:32:22.842706 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:22.842693    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c01c1a4e-7d4d-45ca-b92e-c1cdd9013059-openshift-service-ca\") pod \"maas-api-e2e-inf-b-07e9c8-f548454dc-lsx4x\" (UID: \"c01c1a4e-7d4d-45ca-b92e-c1cdd9013059\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-07e9c8-f548454dc-lsx4x"
Oct 30 22:32:22.842795 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:32:22.842775    5678 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-07e9c8-serving-cert: secret "maas-api-e2e-inf-b-07e9c8-serving-cert" not found
Oct 30 22:32:22.842897 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:32:22.842885    5678 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c01c1a4e-7d4d-45ca-b92e-c1cdd9013059-maas-api-tls podName:c01c1a4e-7d4d-45ca-b92e-c1cdd9013059 nodeName:}" failed. No retries permitted until 2026-10-30 22:32:23.342863761 +0000 UTC m=+2520.947171316 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c01c1a4e-7d4d-45ca-b92e-c1cdd9013059-maas-api-tls") pod "maas-api-e2e-inf-b-07e9c8-f548454dc-lsx4x" (UID: "c01c1a4e-7d4d-45ca-b92e-c1cdd9013059") : secret "maas-api-e2e-inf-b-07e9c8-serving-cert" not found
Oct 30 22:32:22.843346 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:22.843323    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c01c1a4e-7d4d-45ca-b92e-c1cdd9013059-openshift-service-ca\") pod \"maas-api-e2e-inf-b-07e9c8-f548454dc-lsx4x\" (UID: \"c01c1a4e-7d4d-45ca-b92e-c1cdd9013059\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-07e9c8-f548454dc-lsx4x"
Oct 30 22:32:22.845015 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:22.844991    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c01c1a4e-7d4d-45ca-b92e-c1cdd9013059-metrics-certs\") pod \"maas-api-e2e-inf-b-07e9c8-f548454dc-lsx4x\" (UID: \"c01c1a4e-7d4d-45ca-b92e-c1cdd9013059\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-07e9c8-f548454dc-lsx4x"
Oct 30 22:32:22.904675 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:22.904623    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rvs8\" (UniqueName: \"kubernetes.io/projected/c01c1a4e-7d4d-45ca-b92e-c1cdd9013059-kube-api-access-5rvs8\") pod \"maas-api-e2e-inf-b-07e9c8-f548454dc-lsx4x\" (UID: \"c01c1a4e-7d4d-45ca-b92e-c1cdd9013059\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-07e9c8-f548454dc-lsx4x"
Oct 30 22:32:23.348972 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:23.348933    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c01c1a4e-7d4d-45ca-b92e-c1cdd9013059-maas-api-tls\") pod \"maas-api-e2e-inf-b-07e9c8-f548454dc-lsx4x\" (UID: \"c01c1a4e-7d4d-45ca-b92e-c1cdd9013059\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-07e9c8-f548454dc-lsx4x"
Oct 30 22:32:23.351290 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:23.351262    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c01c1a4e-7d4d-45ca-b92e-c1cdd9013059-maas-api-tls\") pod \"maas-api-e2e-inf-b-07e9c8-f548454dc-lsx4x\" (UID: \"c01c1a4e-7d4d-45ca-b92e-c1cdd9013059\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-07e9c8-f548454dc-lsx4x"
Oct 30 22:32:23.585938 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:23.585909    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-07e9c8-f548454dc-lsx4x"
Oct 30 22:32:23.974006 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:23.973924    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-07e9c8-f548454dc-lsx4x"]
Oct 30 22:32:23.980789 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:32:23.980761    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc01c1a4e_7d4d_45ca_b92e_c1cdd9013059.slice/crio-1409363a486fc91851091cf585ce4455d76961fdd2475860bc814c492f593d8f WatchSource:0}: Error finding container 1409363a486fc91851091cf585ce4455d76961fdd2475860bc814c492f593d8f: Status 404 returned error can't find the container with id 1409363a486fc91851091cf585ce4455d76961fdd2475860bc814c492f593d8f
Oct 30 22:32:24.538277 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:24.538245    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-07e9c8-f548454dc-lsx4x" event={"ID":"c01c1a4e-7d4d-45ca-b92e-c1cdd9013059","Type":"ContainerStarted","Data":"1409363a486fc91851091cf585ce4455d76961fdd2475860bc814c492f593d8f"}
Oct 30 22:32:25.546649 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:25.546613    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-07e9c8-f548454dc-lsx4x" event={"ID":"c01c1a4e-7d4d-45ca-b92e-c1cdd9013059","Type":"ContainerStarted","Data":"86131e183696784ce2e01813bda8b728058837755e9e5fadd1379fa41bac8e99"}
Oct 30 22:32:25.547127 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:25.546674    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-07e9c8-f548454dc-lsx4x"
Oct 30 22:32:25.652901 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:25.652865    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-gnkmq"]
Oct 30 22:32:25.653085 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:25.653064    5678 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-gnkmq" podUID="58723455-ca0f-4dbe-ac85-325c6aa5d740" containerName="istio-proxy" containerID="cri-o://09be238885d49a245bc397a3745dcf18a77fdf3e1b7d70ed614deb840b32d988" gracePeriod=30
Oct 30 22:32:25.713444 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:25.713368    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-07e9c8-f548454dc-lsx4x" podStartSLOduration=2.760477536 podStartE2EDuration="3.71335566s" totalImagesPullingTime="952.878124ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:32:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:32:25.712579323 +0000 UTC m=+2523.316886876" watchObservedRunningTime="2026-10-30 22:32:25.71335566 +0000 UTC m=+2523.317663227"
Oct 30 22:32:30.923786 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:30.923727    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-gnkmq"
Oct 30 22:32:31.050227 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:31.050196    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58723455-ca0f-4dbe-ac85-325c6aa5d740-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58723455-ca0f-4dbe-ac85-325c6aa5d740-workload-certs\") pod \"58723455-ca0f-4dbe-ac85-325c6aa5d740\" (UID: \"58723455-ca0f-4dbe-ac85-325c6aa5d740\") "
Oct 30 22:32:31.050437 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:31.050294    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58723455-ca0f-4dbe-ac85-325c6aa5d740-kube-api-access-994tj\" (UniqueName: \"kubernetes.io/projected/58723455-ca0f-4dbe-ac85-325c6aa5d740-kube-api-access-994tj\") pod \"58723455-ca0f-4dbe-ac85-325c6aa5d740\" (UID: \"58723455-ca0f-4dbe-ac85-325c6aa5d740\") "
Oct 30 22:32:31.050437 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:31.050357    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/58723455-ca0f-4dbe-ac85-325c6aa5d740-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58723455-ca0f-4dbe-ac85-325c6aa5d740-istio-podinfo\") pod \"58723455-ca0f-4dbe-ac85-325c6aa5d740\" (UID: \"58723455-ca0f-4dbe-ac85-325c6aa5d740\") "
Oct 30 22:32:31.050578 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:31.050438    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58723455-ca0f-4dbe-ac85-325c6aa5d740-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58723455-ca0f-4dbe-ac85-325c6aa5d740-istiod-ca-cert\") pod \"58723455-ca0f-4dbe-ac85-325c6aa5d740\" (UID: \"58723455-ca0f-4dbe-ac85-325c6aa5d740\") "
Oct 30 22:32:31.050578 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:31.050469    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58723455-ca0f-4dbe-ac85-325c6aa5d740-istio-token\" (UniqueName: \"kubernetes.io/projected/58723455-ca0f-4dbe-ac85-325c6aa5d740-istio-token\") pod \"58723455-ca0f-4dbe-ac85-325c6aa5d740\" (UID: \"58723455-ca0f-4dbe-ac85-325c6aa5d740\") "
Oct 30 22:32:31.050578 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:31.050514    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58723455-ca0f-4dbe-ac85-325c6aa5d740-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58723455-ca0f-4dbe-ac85-325c6aa5d740-workload-socket\") pod \"58723455-ca0f-4dbe-ac85-325c6aa5d740\" (UID: \"58723455-ca0f-4dbe-ac85-325c6aa5d740\") "
Oct 30 22:32:31.050578 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:31.050522    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58723455-ca0f-4dbe-ac85-325c6aa5d740-workload-certs" pod "58723455-ca0f-4dbe-ac85-325c6aa5d740" (UID: "58723455-ca0f-4dbe-ac85-325c6aa5d740"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:32:31.050805 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:31.050743    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58723455-ca0f-4dbe-ac85-325c6aa5d740-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58723455-ca0f-4dbe-ac85-325c6aa5d740-credential-socket\") pod \"58723455-ca0f-4dbe-ac85-325c6aa5d740\" (UID: \"58723455-ca0f-4dbe-ac85-325c6aa5d740\") "
Oct 30 22:32:31.050805 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:31.050757    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58723455-ca0f-4dbe-ac85-325c6aa5d740-workload-socket" pod "58723455-ca0f-4dbe-ac85-325c6aa5d740" (UID: "58723455-ca0f-4dbe-ac85-325c6aa5d740"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:32:31.050935 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:31.050839    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58723455-ca0f-4dbe-ac85-325c6aa5d740-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58723455-ca0f-4dbe-ac85-325c6aa5d740-istio-data\") pod \"58723455-ca0f-4dbe-ac85-325c6aa5d740\" (UID: \"58723455-ca0f-4dbe-ac85-325c6aa5d740\") "
Oct 30 22:32:31.050935 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:31.050848    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58723455-ca0f-4dbe-ac85-325c6aa5d740-istiod-ca-cert" pod "58723455-ca0f-4dbe-ac85-325c6aa5d740" (UID: "58723455-ca0f-4dbe-ac85-325c6aa5d740"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:32:31.050935 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:31.050869    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58723455-ca0f-4dbe-ac85-325c6aa5d740-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58723455-ca0f-4dbe-ac85-325c6aa5d740-istio-envoy\") pod \"58723455-ca0f-4dbe-ac85-325c6aa5d740\" (UID: \"58723455-ca0f-4dbe-ac85-325c6aa5d740\") "
Oct 30 22:32:31.051095 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:31.050997    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58723455-ca0f-4dbe-ac85-325c6aa5d740-credential-socket" pod "58723455-ca0f-4dbe-ac85-325c6aa5d740" (UID: "58723455-ca0f-4dbe-ac85-325c6aa5d740"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:32:31.051502 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:31.051362    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58723455-ca0f-4dbe-ac85-325c6aa5d740-istio-data" pod "58723455-ca0f-4dbe-ac85-325c6aa5d740" (UID: "58723455-ca0f-4dbe-ac85-325c6aa5d740"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:32:31.051502 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:31.051382    5678 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58723455-ca0f-4dbe-ac85-325c6aa5d740-workload-certs\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:32:31.051502 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:31.051438    5678 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58723455-ca0f-4dbe-ac85-325c6aa5d740-istiod-ca-cert\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:32:31.051502 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:31.051459    5678 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58723455-ca0f-4dbe-ac85-325c6aa5d740-workload-socket\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:32:31.051502 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:31.051474    5678 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58723455-ca0f-4dbe-ac85-325c6aa5d740-credential-socket\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:32:31.052810 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:31.052783    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/58723455-ca0f-4dbe-ac85-325c6aa5d740-istio-podinfo" pod "58723455-ca0f-4dbe-ac85-325c6aa5d740" (UID: "58723455-ca0f-4dbe-ac85-325c6aa5d740"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:32:31.052810 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:31.052798    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58723455-ca0f-4dbe-ac85-325c6aa5d740-istio-token" pod "58723455-ca0f-4dbe-ac85-325c6aa5d740" (UID: "58723455-ca0f-4dbe-ac85-325c6aa5d740"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:32:31.052977 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:31.052827    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58723455-ca0f-4dbe-ac85-325c6aa5d740-kube-api-access-994tj" pod "58723455-ca0f-4dbe-ac85-325c6aa5d740" (UID: "58723455-ca0f-4dbe-ac85-325c6aa5d740"). InnerVolumeSpecName "kube-api-access-994tj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:32:31.053177 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:31.053163    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58723455-ca0f-4dbe-ac85-325c6aa5d740-istio-envoy" pod "58723455-ca0f-4dbe-ac85-325c6aa5d740" (UID: "58723455-ca0f-4dbe-ac85-325c6aa5d740"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:32:31.153000 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:31.152920    5678 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58723455-ca0f-4dbe-ac85-325c6aa5d740-istio-podinfo\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:32:31.153000 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:31.152959    5678 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58723455-ca0f-4dbe-ac85-325c6aa5d740-istio-token\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:32:31.153000 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:31.152974    5678 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58723455-ca0f-4dbe-ac85-325c6aa5d740-istio-data\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:32:31.153000 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:31.152987    5678 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58723455-ca0f-4dbe-ac85-325c6aa5d740-istio-envoy\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:32:31.153264 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:31.153001    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-994tj\" (UniqueName: \"kubernetes.io/projected/58723455-ca0f-4dbe-ac85-325c6aa5d740-kube-api-access-994tj\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:32:31.560638 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:31.560612    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-07e9c8-f548454dc-lsx4x"
Oct 30 22:32:31.585893 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:31.585864    5678 generic.go:356] "Generic (PLEG): container finished" podID="58723455-ca0f-4dbe-ac85-325c6aa5d740" containerID="09be238885d49a245bc397a3745dcf18a77fdf3e1b7d70ed614deb840b32d988" exitCode=0
Oct 30 22:32:31.586023 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:31.585904    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-gnkmq"
Oct 30 22:32:31.586023 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:31.585946    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-gnkmq" event={"ID":"58723455-ca0f-4dbe-ac85-325c6aa5d740","Type":"ContainerDied","Data":"09be238885d49a245bc397a3745dcf18a77fdf3e1b7d70ed614deb840b32d988"}
Oct 30 22:32:31.586023 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:31.586001    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-gnkmq" event={"ID":"58723455-ca0f-4dbe-ac85-325c6aa5d740","Type":"ContainerDied","Data":"6324d6dfa23d3a2bf940e6fe26e9f77b7f4d28d9fe37e3e1c956418197ebb2d7"}
Oct 30 22:32:31.586142 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:31.586029    5678 scope.go:122] "RemoveContainer" containerID="09be238885d49a245bc397a3745dcf18a77fdf3e1b7d70ed614deb840b32d988"
Oct 30 22:32:31.598484 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:31.598464    5678 scope.go:122] "RemoveContainer" containerID="09be238885d49a245bc397a3745dcf18a77fdf3e1b7d70ed614deb840b32d988"
Oct 30 22:32:31.598893 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:32:31.598869    5678 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09be238885d49a245bc397a3745dcf18a77fdf3e1b7d70ed614deb840b32d988\": container with ID starting with 09be238885d49a245bc397a3745dcf18a77fdf3e1b7d70ed614deb840b32d988 not found: ID does not exist" containerID="09be238885d49a245bc397a3745dcf18a77fdf3e1b7d70ed614deb840b32d988"
Oct 30 22:32:31.598995 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:31.598899    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09be238885d49a245bc397a3745dcf18a77fdf3e1b7d70ed614deb840b32d988"} err="failed to get container status \"09be238885d49a245bc397a3745dcf18a77fdf3e1b7d70ed614deb840b32d988\": rpc error: code = NotFound desc = could not find container \"09be238885d49a245bc397a3745dcf18a77fdf3e1b7d70ed614deb840b32d988\": container with ID starting with 09be238885d49a245bc397a3745dcf18a77fdf3e1b7d70ed614deb840b32d988 not found: ID does not exist"
Oct 30 22:32:31.800340 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:31.800312    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-gnkmq"]
Oct 30 22:32:31.811588 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:31.811515    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-gnkmq"]
Oct 30 22:32:32.523892 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:32.523863    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58723455-ca0f-4dbe-ac85-325c6aa5d740" path="/var/lib/kubelet/pods/58723455-ca0f-4dbe-ac85-325c6aa5d740/volumes"
Oct 30 22:32:40.572042 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:32:40.572004    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 22:32:43.259998 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:43.259961    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-bc714d-7c5b5598f-phvl6"]
Oct 30 22:32:43.364949 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:43.364910    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-bc714d-57765d6599-txzbm"]
Oct 30 22:32:48.260767 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:48.260720    5678 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-bc714d-7c5b5598f-phvl6" podUID="5f9fdabd-6c3b-4bbe-b1ea-ef2159076fc1" containerName="payload-processing" containerID="cri-o://fd76d1901f436997d6f834a701e64501c1344ff94f4b4466632872e5b535dc00" gracePeriod=25
Oct 30 22:32:48.366033 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:48.365994    5678 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-bc714d-57765d6599-txzbm" podUID="6b45b1a6-24c2-4170-b914-85c2a9a5aab8" containerName="payload-pre-processing" containerID="cri-o://5078d8146efb238945600b2bd45ccd3ef4bd2815da0a9fe04112eefa833904c0" gracePeriod=25
Oct 30 22:32:51.556764 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:51.556720    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-506964-5d557d7b77-77m8h"]
Oct 30 22:32:51.557310 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:51.557293    5678 cpu_manager.go:414] "containerMap: removing container" podUID="58723455-ca0f-4dbe-ac85-325c6aa5d740" containerName="istio-proxy"
Oct 30 22:32:51.557310 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:51.557308    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58723455-ca0f-4dbe-ac85-325c6aa5d740" containerName="istio-proxy"
Oct 30 22:32:51.557492 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:51.557388    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="707b0553-ce35-4de0-ba4a-fffa4adfc6ee" containerName="istio-proxy"
Oct 30 22:32:51.557492 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:51.557412    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="58723455-ca0f-4dbe-ac85-325c6aa5d740" containerName="istio-proxy"
Oct 30 22:32:51.561148 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:51.561124    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-506964-5d557d7b77-77m8h"
Oct 30 22:32:51.616309 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:51.616285    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-506964-serving-cert\""
Oct 30 22:32:51.623697 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:51.623669    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfrgk\" (UniqueName: \"kubernetes.io/projected/699c4653-4da2-4b6f-965e-d34b632a8340-kube-api-access-vfrgk\") pod \"maas-api-e2e-inf-b-506964-5d557d7b77-77m8h\" (UID: \"699c4653-4da2-4b6f-965e-d34b632a8340\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-506964-5d557d7b77-77m8h"
Oct 30 22:32:51.623897 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:51.623864    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/699c4653-4da2-4b6f-965e-d34b632a8340-maas-api-tls\") pod \"maas-api-e2e-inf-b-506964-5d557d7b77-77m8h\" (UID: \"699c4653-4da2-4b6f-965e-d34b632a8340\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-506964-5d557d7b77-77m8h"
Oct 30 22:32:51.624024 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:51.623987    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/699c4653-4da2-4b6f-965e-d34b632a8340-openshift-service-ca\") pod \"maas-api-e2e-inf-b-506964-5d557d7b77-77m8h\" (UID: \"699c4653-4da2-4b6f-965e-d34b632a8340\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-506964-5d557d7b77-77m8h"
Oct 30 22:32:51.624109 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:51.624083    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/699c4653-4da2-4b6f-965e-d34b632a8340-metrics-certs\") pod \"maas-api-e2e-inf-b-506964-5d557d7b77-77m8h\" (UID: \"699c4653-4da2-4b6f-965e-d34b632a8340\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-506964-5d557d7b77-77m8h"
Oct 30 22:32:51.668559 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:51.668527    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-506964-5d557d7b77-77m8h"]
Oct 30 22:32:51.725072 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:51.725036    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfrgk\" (UniqueName: \"kubernetes.io/projected/699c4653-4da2-4b6f-965e-d34b632a8340-kube-api-access-vfrgk\") pod \"maas-api-e2e-inf-b-506964-5d557d7b77-77m8h\" (UID: \"699c4653-4da2-4b6f-965e-d34b632a8340\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-506964-5d557d7b77-77m8h"
Oct 30 22:32:51.725272 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:51.725114    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/699c4653-4da2-4b6f-965e-d34b632a8340-maas-api-tls\") pod \"maas-api-e2e-inf-b-506964-5d557d7b77-77m8h\" (UID: \"699c4653-4da2-4b6f-965e-d34b632a8340\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-506964-5d557d7b77-77m8h"
Oct 30 22:32:51.725272 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:51.725213    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/699c4653-4da2-4b6f-965e-d34b632a8340-openshift-service-ca\") pod \"maas-api-e2e-inf-b-506964-5d557d7b77-77m8h\" (UID: \"699c4653-4da2-4b6f-965e-d34b632a8340\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-506964-5d557d7b77-77m8h"
Oct 30 22:32:51.725272 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:32:51.725259    5678 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-506964-serving-cert: secret "maas-api-e2e-inf-b-506964-serving-cert" not found
Oct 30 22:32:51.725455 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:51.725285    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/699c4653-4da2-4b6f-965e-d34b632a8340-metrics-certs\") pod \"maas-api-e2e-inf-b-506964-5d557d7b77-77m8h\" (UID: \"699c4653-4da2-4b6f-965e-d34b632a8340\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-506964-5d557d7b77-77m8h"
Oct 30 22:32:51.725455 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:32:51.725348    5678 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/699c4653-4da2-4b6f-965e-d34b632a8340-maas-api-tls podName:699c4653-4da2-4b6f-965e-d34b632a8340 nodeName:}" failed. No retries permitted until 2026-10-30 22:32:52.225325033 +0000 UTC m=+2549.829632585 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/699c4653-4da2-4b6f-965e-d34b632a8340-maas-api-tls") pod "maas-api-e2e-inf-b-506964-5d557d7b77-77m8h" (UID: "699c4653-4da2-4b6f-965e-d34b632a8340") : secret "maas-api-e2e-inf-b-506964-serving-cert" not found
Oct 30 22:32:51.726588 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:51.726562    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/699c4653-4da2-4b6f-965e-d34b632a8340-openshift-service-ca\") pod \"maas-api-e2e-inf-b-506964-5d557d7b77-77m8h\" (UID: \"699c4653-4da2-4b6f-965e-d34b632a8340\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-506964-5d557d7b77-77m8h"
Oct 30 22:32:51.728817 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:51.728791    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/699c4653-4da2-4b6f-965e-d34b632a8340-metrics-certs\") pod \"maas-api-e2e-inf-b-506964-5d557d7b77-77m8h\" (UID: \"699c4653-4da2-4b6f-965e-d34b632a8340\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-506964-5d557d7b77-77m8h"
Oct 30 22:32:51.796298 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:51.796265    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfrgk\" (UniqueName: \"kubernetes.io/projected/699c4653-4da2-4b6f-965e-d34b632a8340-kube-api-access-vfrgk\") pod \"maas-api-e2e-inf-b-506964-5d557d7b77-77m8h\" (UID: \"699c4653-4da2-4b6f-965e-d34b632a8340\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-506964-5d557d7b77-77m8h"
Oct 30 22:32:52.067780 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:52.067738    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-bc714d-57765d6599-txzbm" podUID="6b45b1a6-24c2-4170-b914-85c2a9a5aab8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:32:52.067970 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:52.067754    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-bc714d-7c5b5598f-phvl6" podUID="5f9fdabd-6c3b-4bbe-b1ea-ef2159076fc1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:32:52.233373 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:52.233341    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/699c4653-4da2-4b6f-965e-d34b632a8340-maas-api-tls\") pod \"maas-api-e2e-inf-b-506964-5d557d7b77-77m8h\" (UID: \"699c4653-4da2-4b6f-965e-d34b632a8340\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-506964-5d557d7b77-77m8h"
Oct 30 22:32:52.235735 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:52.235707    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/699c4653-4da2-4b6f-965e-d34b632a8340-maas-api-tls\") pod \"maas-api-e2e-inf-b-506964-5d557d7b77-77m8h\" (UID: \"699c4653-4da2-4b6f-965e-d34b632a8340\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-506964-5d557d7b77-77m8h"
Oct 30 22:32:52.476325 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:52.476291    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-506964-5d557d7b77-77m8h"
Oct 30 22:32:52.740877 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:52.740810    5678 generic.go:356] "Generic (PLEG): container finished" podID="f8eba014-2735-4189-a982-e33413ce8e4b" containerID="a5e150eb8bd120fca6311562b9ea5e1f23c79a3670a3aa5f575f15df9856d9a1" exitCode=0
Oct 30 22:32:52.741289 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:52.740879    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-07e9c8-openshift-default-5944f97d49-m76c9" event={"ID":"f8eba014-2735-4189-a982-e33413ce8e4b","Type":"ContainerDied","Data":"a5e150eb8bd120fca6311562b9ea5e1f23c79a3670a3aa5f575f15df9856d9a1"}
Oct 30 22:32:52.741289 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:52.741285    5678 scope.go:122] "RemoveContainer" containerID="a5e150eb8bd120fca6311562b9ea5e1f23c79a3670a3aa5f575f15df9856d9a1"
Oct 30 22:32:52.742962 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:52.742931    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:32:52.743096 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:52.743026    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:32:52.743096 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:52.743065    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:32:52.902040 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:52.902009    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-506964-5d557d7b77-77m8h"]
Oct 30 22:32:52.908918 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:32:52.908893    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod699c4653_4da2_4b6f_965e_d34b632a8340.slice/crio-ee6be2ce0f37db8201f8edbbf96c238cd3b9bf479083d6d3534d50d9e877ce38 WatchSource:0}: Error finding container ee6be2ce0f37db8201f8edbbf96c238cd3b9bf479083d6d3534d50d9e877ce38: Status 404 returned error can't find the container with id ee6be2ce0f37db8201f8edbbf96c238cd3b9bf479083d6d3534d50d9e877ce38
Oct 30 22:32:53.754605 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:53.754562    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-506964-5d557d7b77-77m8h" event={"ID":"699c4653-4da2-4b6f-965e-d34b632a8340","Type":"ContainerStarted","Data":"ee6be2ce0f37db8201f8edbbf96c238cd3b9bf479083d6d3534d50d9e877ce38"}
Oct 30 22:32:53.756608 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:53.756583    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-07e9c8-openshift-default-5944f97d49-m76c9" event={"ID":"f8eba014-2735-4189-a982-e33413ce8e4b","Type":"ContainerStarted","Data":"38afa7c023ddcdbab4c782283d3887435ad57de709f00e5888f0dcd9fd7e9995"}
Oct 30 22:32:54.210987 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:54.210954    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-07e9c8-openshift-default-5944f97d49-m76c9"
Oct 30 22:32:54.212245 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:54.212217    5678 patch_prober.go:29] interesting pod/e2e-inf-b-07e9c8-openshift-default-5944f97d49-m76c9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body=
Oct 30 22:32:54.212354 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:54.212264    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-07e9c8-openshift-default-5944f97d49-m76c9" podUID="f8eba014-2735-4189-a982-e33413ce8e4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused"
Oct 30 22:32:54.764460 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:54.764428    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-506964-5d557d7b77-77m8h" event={"ID":"699c4653-4da2-4b6f-965e-d34b632a8340","Type":"ContainerStarted","Data":"5b70af7c2321d75355369d6883cc2824aa369ca459533f36e200cf462610883a"}
Oct 30 22:32:54.764953 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:54.764524    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-506964-5d557d7b77-77m8h"
Oct 30 22:32:54.884653 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:54.884596    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-506964-5d557d7b77-77m8h" podStartSLOduration=2.942886462 podStartE2EDuration="3.884583653s" totalImagesPullingTime="941.697191ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:32:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:32:54.882242157 +0000 UTC m=+2552.486549710" watchObservedRunningTime="2026-10-30 22:32:54.884583653 +0000 UTC m=+2552.488891206"
Oct 30 22:32:55.211084 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:55.211051    5678 patch_prober.go:29] interesting pod/e2e-inf-b-07e9c8-openshift-default-5944f97d49-m76c9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body=
Oct 30 22:32:55.211265 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:55.211100    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-07e9c8-openshift-default-5944f97d49-m76c9" podUID="f8eba014-2735-4189-a982-e33413ce8e4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused"
Oct 30 22:32:56.103492 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:56.103461    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-07e9c8-openshift-default-5944f97d49-m76c9"
Oct 30 22:32:56.210869 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:56.210837    5678 patch_prober.go:29] interesting pod/e2e-inf-b-07e9c8-openshift-default-5944f97d49-m76c9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body=
Oct 30 22:32:56.211041 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:56.210886    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-07e9c8-openshift-default-5944f97d49-m76c9" podUID="f8eba014-2735-4189-a982-e33413ce8e4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused"
Oct 30 22:32:57.211266 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:57.211221    5678 patch_prober.go:29] interesting pod/e2e-inf-b-07e9c8-openshift-default-5944f97d49-m76c9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body=
Oct 30 22:32:57.211708 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:57.211279    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-07e9c8-openshift-default-5944f97d49-m76c9" podUID="f8eba014-2735-4189-a982-e33413ce8e4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused"
Oct 30 22:32:58.210939 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:58.210903    5678 patch_prober.go:29] interesting pod/e2e-inf-b-07e9c8-openshift-default-5944f97d49-m76c9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body=
Oct 30 22:32:58.211122 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:58.210953    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-07e9c8-openshift-default-5944f97d49-m76c9" podUID="f8eba014-2735-4189-a982-e33413ce8e4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused"
Oct 30 22:32:59.275780 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:59.275746    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-07e9c8-openshift-default-5944f97d49-m76c9"
Oct 30 22:32:59.277451 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:32:59.277427    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-07e9c8-openshift-default-5944f97d49-m76c9"
Oct 30 22:33:00.777348 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:33:00.777319    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-506964-5d557d7b77-77m8h"
Oct 30 22:33:02.067326 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:33:02.067290    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-bc714d-57765d6599-txzbm" podUID="6b45b1a6-24c2-4170-b914-85c2a9a5aab8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:33:02.067745 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:33:02.067476    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-bc714d-7c5b5598f-phvl6" podUID="5f9fdabd-6c3b-4bbe-b1ea-ef2159076fc1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:33:08.544380 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:33:08.544347    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-bc714d-7c5b5598f-phvl6"
Oct 30 22:33:08.563427 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:33:08.563369    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f9fdabd-6c3b-4bbe-b1ea-ef2159076fc1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f9fdabd-6c3b-4bbe-b1ea-ef2159076fc1-plugins-config-volume\") pod \"5f9fdabd-6c3b-4bbe-b1ea-ef2159076fc1\" (UID: \"5f9fdabd-6c3b-4bbe-b1ea-ef2159076fc1\") "
Oct 30 22:33:08.563594 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:33:08.563572    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f9fdabd-6c3b-4bbe-b1ea-ef2159076fc1-kube-api-access-kdn7q\" (UniqueName: \"kubernetes.io/projected/5f9fdabd-6c3b-4bbe-b1ea-ef2159076fc1-kube-api-access-kdn7q\") pod \"5f9fdabd-6c3b-4bbe-b1ea-ef2159076fc1\" (UID: \"5f9fdabd-6c3b-4bbe-b1ea-ef2159076fc1\") "
Oct 30 22:33:08.563842 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:33:08.563814    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f9fdabd-6c3b-4bbe-b1ea-ef2159076fc1-plugins-config-volume" pod "5f9fdabd-6c3b-4bbe-b1ea-ef2159076fc1" (UID: "5f9fdabd-6c3b-4bbe-b1ea-ef2159076fc1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:33:08.564157 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:33:08.564132    5678 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f9fdabd-6c3b-4bbe-b1ea-ef2159076fc1-plugins-config-volume\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:33:08.566492 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:33:08.566459    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f9fdabd-6c3b-4bbe-b1ea-ef2159076fc1-kube-api-access-kdn7q" pod "5f9fdabd-6c3b-4bbe-b1ea-ef2159076fc1" (UID: "5f9fdabd-6c3b-4bbe-b1ea-ef2159076fc1"). InnerVolumeSpecName "kube-api-access-kdn7q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:33:08.624918 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:33:08.624892    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-bc714d-57765d6599-txzbm"
Oct 30 22:33:08.664663 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:33:08.664639    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b45b1a6-24c2-4170-b914-85c2a9a5aab8-kube-api-access-8gm97\" (UniqueName: \"kubernetes.io/projected/6b45b1a6-24c2-4170-b914-85c2a9a5aab8-kube-api-access-8gm97\") pod \"6b45b1a6-24c2-4170-b914-85c2a9a5aab8\" (UID: \"6b45b1a6-24c2-4170-b914-85c2a9a5aab8\") "
Oct 30 22:33:08.664847 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:33:08.664832    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b45b1a6-24c2-4170-b914-85c2a9a5aab8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b45b1a6-24c2-4170-b914-85c2a9a5aab8-plugins-config-volume\") pod \"6b45b1a6-24c2-4170-b914-85c2a9a5aab8\" (UID: \"6b45b1a6-24c2-4170-b914-85c2a9a5aab8\") "
Oct 30 22:33:08.665272 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:33:08.665243    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b45b1a6-24c2-4170-b914-85c2a9a5aab8-plugins-config-volume" pod "6b45b1a6-24c2-4170-b914-85c2a9a5aab8" (UID: "6b45b1a6-24c2-4170-b914-85c2a9a5aab8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:33:08.665364 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:33:08.665283    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdn7q\" (UniqueName: \"kubernetes.io/projected/5f9fdabd-6c3b-4bbe-b1ea-ef2159076fc1-kube-api-access-kdn7q\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:33:08.666661 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:33:08.666641    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b45b1a6-24c2-4170-b914-85c2a9a5aab8-kube-api-access-8gm97" pod "6b45b1a6-24c2-4170-b914-85c2a9a5aab8" (UID: "6b45b1a6-24c2-4170-b914-85c2a9a5aab8"). InnerVolumeSpecName "kube-api-access-8gm97". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:33:08.766467 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:33:08.766436    5678 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b45b1a6-24c2-4170-b914-85c2a9a5aab8-plugins-config-volume\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:33:08.766467 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:33:08.766466    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gm97\" (UniqueName: \"kubernetes.io/projected/6b45b1a6-24c2-4170-b914-85c2a9a5aab8-kube-api-access-8gm97\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:33:08.861428 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:33:08.861321    5678 generic.go:356] "Generic (PLEG): container finished" podID="5f9fdabd-6c3b-4bbe-b1ea-ef2159076fc1" containerID="fd76d1901f436997d6f834a701e64501c1344ff94f4b4466632872e5b535dc00" exitCode=0
Oct 30 22:33:08.861428 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:33:08.861364    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-bc714d-7c5b5598f-phvl6"
Oct 30 22:33:08.861428 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:33:08.861418    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-bc714d-7c5b5598f-phvl6" event={"ID":"5f9fdabd-6c3b-4bbe-b1ea-ef2159076fc1","Type":"ContainerDied","Data":"fd76d1901f436997d6f834a701e64501c1344ff94f4b4466632872e5b535dc00"}
Oct 30 22:33:08.861774 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:33:08.861473    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-bc714d-7c5b5598f-phvl6" event={"ID":"5f9fdabd-6c3b-4bbe-b1ea-ef2159076fc1","Type":"ContainerDied","Data":"59c0dfd9b78853b2ad679a444401d735783b2813e727b32f75bbbfe70c51d180"}
Oct 30 22:33:08.861774 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:33:08.861504    5678 scope.go:122] "RemoveContainer" containerID="fd76d1901f436997d6f834a701e64501c1344ff94f4b4466632872e5b535dc00"
Oct 30 22:33:08.862848 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:33:08.862822    5678 generic.go:356] "Generic (PLEG): container finished" podID="6b45b1a6-24c2-4170-b914-85c2a9a5aab8" containerID="5078d8146efb238945600b2bd45ccd3ef4bd2815da0a9fe04112eefa833904c0" exitCode=0
Oct 30 22:33:08.862983 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:33:08.862873    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-bc714d-57765d6599-txzbm"
Oct 30 22:33:08.862983 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:33:08.862909    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-bc714d-57765d6599-txzbm" event={"ID":"6b45b1a6-24c2-4170-b914-85c2a9a5aab8","Type":"ContainerDied","Data":"5078d8146efb238945600b2bd45ccd3ef4bd2815da0a9fe04112eefa833904c0"}
Oct 30 22:33:08.862983 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:33:08.862960    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-bc714d-57765d6599-txzbm" event={"ID":"6b45b1a6-24c2-4170-b914-85c2a9a5aab8","Type":"ContainerDied","Data":"6dbcac47238d3ec8b6313a06671609af6b69078e327f7ba0535cbe8eb44d5d01"}
Oct 30 22:33:08.872372 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:33:08.872349    5678 scope.go:122] "RemoveContainer" containerID="fd76d1901f436997d6f834a701e64501c1344ff94f4b4466632872e5b535dc00"
Oct 30 22:33:08.872683 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:33:08.872661    5678 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd76d1901f436997d6f834a701e64501c1344ff94f4b4466632872e5b535dc00\": container with ID starting with fd76d1901f436997d6f834a701e64501c1344ff94f4b4466632872e5b535dc00 not found: ID does not exist" containerID="fd76d1901f436997d6f834a701e64501c1344ff94f4b4466632872e5b535dc00"
Oct 30 22:33:08.872761 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:33:08.872692    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd76d1901f436997d6f834a701e64501c1344ff94f4b4466632872e5b535dc00"} err="failed to get container status \"fd76d1901f436997d6f834a701e64501c1344ff94f4b4466632872e5b535dc00\": rpc error: code = NotFound desc = could not find container \"fd76d1901f436997d6f834a701e64501c1344ff94f4b4466632872e5b535dc00\": container with ID starting with fd76d1901f436997d6f834a701e64501c1344ff94f4b4466632872e5b535dc00 not found: ID does not exist"
Oct 30 22:33:08.872761 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:33:08.872715    5678 scope.go:122] "RemoveContainer" containerID="5078d8146efb238945600b2bd45ccd3ef4bd2815da0a9fe04112eefa833904c0"
Oct 30 22:33:08.882421 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:33:08.882379    5678 scope.go:122] "RemoveContainer" containerID="5078d8146efb238945600b2bd45ccd3ef4bd2815da0a9fe04112eefa833904c0"
Oct 30 22:33:08.882688 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:33:08.882669    5678 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5078d8146efb238945600b2bd45ccd3ef4bd2815da0a9fe04112eefa833904c0\": container with ID starting with 5078d8146efb238945600b2bd45ccd3ef4bd2815da0a9fe04112eefa833904c0 not found: ID does not exist" containerID="5078d8146efb238945600b2bd45ccd3ef4bd2815da0a9fe04112eefa833904c0"
Oct 30 22:33:08.882757 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:33:08.882696    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5078d8146efb238945600b2bd45ccd3ef4bd2815da0a9fe04112eefa833904c0"} err="failed to get container status \"5078d8146efb238945600b2bd45ccd3ef4bd2815da0a9fe04112eefa833904c0\": rpc error: code = NotFound desc = could not find container \"5078d8146efb238945600b2bd45ccd3ef4bd2815da0a9fe04112eefa833904c0\": container with ID starting with 5078d8146efb238945600b2bd45ccd3ef4bd2815da0a9fe04112eefa833904c0 not found: ID does not exist"
Oct 30 22:33:09.022417 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:33:09.022375    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-bc714d-57765d6599-txzbm"]
Oct 30 22:33:09.026490 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:33:09.026465    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-bc714d-57765d6599-txzbm"]
Oct 30 22:33:09.130307 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:33:09.130275    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-bc714d-7c5b5598f-phvl6"]
Oct 30 22:33:09.135054 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:33:09.135031    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-bc714d-7c5b5598f-phvl6"]
Oct 30 22:33:10.522935 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:33:10.522906    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f9fdabd-6c3b-4bbe-b1ea-ef2159076fc1" path="/var/lib/kubelet/pods/5f9fdabd-6c3b-4bbe-b1ea-ef2159076fc1/volumes"
Oct 30 22:33:10.523315 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:33:10.523301    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b45b1a6-24c2-4170-b914-85c2a9a5aab8" path="/var/lib/kubelet/pods/6b45b1a6-24c2-4170-b914-85c2a9a5aab8/volumes"
Oct 30 22:33:10.571291 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:33:10.571256    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 22:33:25.003959 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:33:25.003920    5678 generic.go:356] "Generic (PLEG): container finished" podID="64a0ba20-2130-4df1-b147-28a52a7d0e62" containerID="03a0781f38eb9c11adf9a9b59c0598959af5afd188736fd885086d08384bc326" exitCode=0
Oct 30 22:33:25.004414 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:33:25.004009    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-506964-openshift-default-5bfdbc5849-4qwwq" event={"ID":"64a0ba20-2130-4df1-b147-28a52a7d0e62","Type":"ContainerDied","Data":"03a0781f38eb9c11adf9a9b59c0598959af5afd188736fd885086d08384bc326"}
Oct 30 22:33:25.004496 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:33:25.004480    5678 scope.go:122] "RemoveContainer" containerID="03a0781f38eb9c11adf9a9b59c0598959af5afd188736fd885086d08384bc326"
Oct 30 22:33:25.005960 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:33:25.005934    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:33:25.006043 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:33:25.006002    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:33:25.006043 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:33:25.006035    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:33:26.013162 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:33:26.013124    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-506964-openshift-default-5bfdbc5849-4qwwq" event={"ID":"64a0ba20-2130-4df1-b147-28a52a7d0e62","Type":"ContainerStarted","Data":"2a2a0c368ccd60107ea46ad21cd211e2bdf53734db49f376966c0db909f47426"}
Oct 30 22:33:26.751342 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:33:26.751302    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-506964-openshift-default-5bfdbc5849-4qwwq"
Oct 30 22:33:26.751564 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:33:26.751528    5678 patch_prober.go:29] interesting pod/e2e-inf-b-506964-openshift-default-5bfdbc5849-4qwwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body=
Oct 30 22:33:26.751617 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:33:26.751576    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-506964-openshift-default-5bfdbc5849-4qwwq" podUID="64a0ba20-2130-4df1-b147-28a52a7d0e62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused"
Oct 30 22:33:27.751918 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:33:27.751874    5678 patch_prober.go:29] interesting pod/e2e-inf-b-506964-openshift-default-5bfdbc5849-4qwwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body=
Oct 30 22:33:27.752434 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:33:27.751935    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-506964-openshift-default-5bfdbc5849-4qwwq" podUID="64a0ba20-2130-4df1-b147-28a52a7d0e62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused"
Oct 30 22:33:28.751843 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:33:28.751809    5678 patch_prober.go:29] interesting pod/e2e-inf-b-506964-openshift-default-5bfdbc5849-4qwwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body=
Oct 30 22:33:28.752056 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:33:28.751868    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-506964-openshift-default-5bfdbc5849-4qwwq" podUID="64a0ba20-2130-4df1-b147-28a52a7d0e62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused"
Oct 30 22:33:29.751952 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:33:29.751905    5678 patch_prober.go:29] interesting pod/e2e-inf-b-506964-openshift-default-5bfdbc5849-4qwwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body=
Oct 30 22:33:29.752130 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:33:29.751979    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-506964-openshift-default-5bfdbc5849-4qwwq" podUID="64a0ba20-2130-4df1-b147-28a52a7d0e62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused"
Oct 30 22:33:30.751799 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:33:30.751761    5678 patch_prober.go:29] interesting pod/e2e-inf-b-506964-openshift-default-5bfdbc5849-4qwwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body=
Oct 30 22:33:30.751978 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:33:30.751857    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-506964-openshift-default-5bfdbc5849-4qwwq" podUID="64a0ba20-2130-4df1-b147-28a52a7d0e62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused"
Oct 30 22:33:31.756098 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:33:31.756068    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-506964-openshift-default-5bfdbc5849-4qwwq"
Oct 30 22:33:31.756612 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:33:31.756430    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-506964-openshift-default-5bfdbc5849-4qwwq"
Oct 30 22:33:31.757025 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:33:31.757006    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-506964-openshift-default-5bfdbc5849-4qwwq"
Oct 30 22:33:40.571611 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:33:40.571573    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 22:34:10.572105 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:34:10.572027    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 22:34:11.736954 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:11.736922    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cqvp9"]
Oct 30 22:34:11.737480 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:11.737111    5678 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cqvp9" podUID="4e913a61-bea1-4281-9d5d-cdab5506a864" containerName="istio-proxy" containerID="cri-o://2f6d42261c7f0ce6f5c924ba6332ac7eb03692406eb0200a79b0b374ef17d304" gracePeriod=30
Oct 30 22:34:13.157461 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:13.157427    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-qp9q4"]
Oct 30 22:34:13.157851 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:13.157622    5678 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-qp9q4" podUID="e400e450-98f2-423b-bbc1-478fb88f61a2" containerName="istio-proxy" containerID="cri-o://fdeb0d71791c9066e532453a2f5662b6bd9cea6cdfea15e7c29496a2e09ca469" gracePeriod=30
Oct 30 22:34:13.841560 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:13.841523    5678 patch_prober.go:29] interesting pod/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-qp9q4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:34:13.841732 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:13.841579    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-qp9q4" podUID="e400e450-98f2-423b-bbc1-478fb88f61a2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:34:15.288590 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:15.288555    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-wqqzd"]
Oct 30 22:34:15.289049 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:15.288766    5678 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-wqqzd" podUID="99f99481-73d0-4fb4-9c88-12ee143a6904" containerName="istio-proxy" containerID="cri-o://ba02e73767820b8f4d9f9f6c922d0537157b67d8705891fb420aabb534e608c4" gracePeriod=30
Oct 30 22:34:15.414073 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:15.414039    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-779f32/premium-w0-779f32-kserve-67978797c8-jpkvl"]
Oct 30 22:34:16.998998 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:16.998971    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cqvp9"
Oct 30 22:34:17.172454 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:17.172387    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4e913a61-bea1-4281-9d5d-cdab5506a864-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e913a61-bea1-4281-9d5d-cdab5506a864-istio-podinfo\") pod \"4e913a61-bea1-4281-9d5d-cdab5506a864\" (UID: \"4e913a61-bea1-4281-9d5d-cdab5506a864\") "
Oct 30 22:34:17.172665 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:17.172494    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e913a61-bea1-4281-9d5d-cdab5506a864-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e913a61-bea1-4281-9d5d-cdab5506a864-istio-envoy\") pod \"4e913a61-bea1-4281-9d5d-cdab5506a864\" (UID: \"4e913a61-bea1-4281-9d5d-cdab5506a864\") "
Oct 30 22:34:17.172665 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:17.172555    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e913a61-bea1-4281-9d5d-cdab5506a864-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e913a61-bea1-4281-9d5d-cdab5506a864-credential-socket\") pod \"4e913a61-bea1-4281-9d5d-cdab5506a864\" (UID: \"4e913a61-bea1-4281-9d5d-cdab5506a864\") "
Oct 30 22:34:17.172665 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:17.172599    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e913a61-bea1-4281-9d5d-cdab5506a864-kube-api-access-hczps\" (UniqueName: \"kubernetes.io/projected/4e913a61-bea1-4281-9d5d-cdab5506a864-kube-api-access-hczps\") pod \"4e913a61-bea1-4281-9d5d-cdab5506a864\" (UID: \"4e913a61-bea1-4281-9d5d-cdab5506a864\") "
Oct 30 22:34:17.172859 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:17.172703    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e913a61-bea1-4281-9d5d-cdab5506a864-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e913a61-bea1-4281-9d5d-cdab5506a864-workload-socket\") pod \"4e913a61-bea1-4281-9d5d-cdab5506a864\" (UID: \"4e913a61-bea1-4281-9d5d-cdab5506a864\") "
Oct 30 22:34:17.172859 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:17.172751    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e913a61-bea1-4281-9d5d-cdab5506a864-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e913a61-bea1-4281-9d5d-cdab5506a864-workload-certs\") pod \"4e913a61-bea1-4281-9d5d-cdab5506a864\" (UID: \"4e913a61-bea1-4281-9d5d-cdab5506a864\") "
Oct 30 22:34:17.172859 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:17.172787    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e913a61-bea1-4281-9d5d-cdab5506a864-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e913a61-bea1-4281-9d5d-cdab5506a864-istiod-ca-cert\") pod \"4e913a61-bea1-4281-9d5d-cdab5506a864\" (UID: \"4e913a61-bea1-4281-9d5d-cdab5506a864\") "
Oct 30 22:34:17.172859 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:17.172813    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e913a61-bea1-4281-9d5d-cdab5506a864-istio-token\" (UniqueName: \"kubernetes.io/projected/4e913a61-bea1-4281-9d5d-cdab5506a864-istio-token\") pod \"4e913a61-bea1-4281-9d5d-cdab5506a864\" (UID: \"4e913a61-bea1-4281-9d5d-cdab5506a864\") "
Oct 30 22:34:17.172859 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:17.172857    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e913a61-bea1-4281-9d5d-cdab5506a864-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e913a61-bea1-4281-9d5d-cdab5506a864-istio-data\") pod \"4e913a61-bea1-4281-9d5d-cdab5506a864\" (UID: \"4e913a61-bea1-4281-9d5d-cdab5506a864\") "
Oct 30 22:34:17.173127 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:17.172882    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e913a61-bea1-4281-9d5d-cdab5506a864-credential-socket" pod "4e913a61-bea1-4281-9d5d-cdab5506a864" (UID: "4e913a61-bea1-4281-9d5d-cdab5506a864"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:34:17.173127 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:17.172957    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e913a61-bea1-4281-9d5d-cdab5506a864-workload-socket" pod "4e913a61-bea1-4281-9d5d-cdab5506a864" (UID: "4e913a61-bea1-4281-9d5d-cdab5506a864"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:34:17.173127 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:17.173016    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e913a61-bea1-4281-9d5d-cdab5506a864-workload-certs" pod "4e913a61-bea1-4281-9d5d-cdab5506a864" (UID: "4e913a61-bea1-4281-9d5d-cdab5506a864"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:34:17.173323 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:17.173226    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e913a61-bea1-4281-9d5d-cdab5506a864-istio-data" pod "4e913a61-bea1-4281-9d5d-cdab5506a864" (UID: "4e913a61-bea1-4281-9d5d-cdab5506a864"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:34:17.173323 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:17.173242    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e913a61-bea1-4281-9d5d-cdab5506a864-istiod-ca-cert" pod "4e913a61-bea1-4281-9d5d-cdab5506a864" (UID: "4e913a61-bea1-4281-9d5d-cdab5506a864"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:34:17.173471 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:17.173446    5678 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e913a61-bea1-4281-9d5d-cdab5506a864-credential-socket\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:17.173546 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:17.173468    5678 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e913a61-bea1-4281-9d5d-cdab5506a864-workload-socket\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:17.173546 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:17.173484    5678 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e913a61-bea1-4281-9d5d-cdab5506a864-workload-certs\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:17.173546 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:17.173493    5678 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e913a61-bea1-4281-9d5d-cdab5506a864-istiod-ca-cert\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:17.173546 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:17.173502    5678 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e913a61-bea1-4281-9d5d-cdab5506a864-istio-data\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:17.174930 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:17.174902    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e913a61-bea1-4281-9d5d-cdab5506a864-istio-envoy" pod "4e913a61-bea1-4281-9d5d-cdab5506a864" (UID: "4e913a61-bea1-4281-9d5d-cdab5506a864"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:34:17.175060 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:17.174936    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e913a61-bea1-4281-9d5d-cdab5506a864-kube-api-access-hczps" pod "4e913a61-bea1-4281-9d5d-cdab5506a864" (UID: "4e913a61-bea1-4281-9d5d-cdab5506a864"). InnerVolumeSpecName "kube-api-access-hczps". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:34:17.175213 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:17.175197    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4e913a61-bea1-4281-9d5d-cdab5506a864-istio-podinfo" pod "4e913a61-bea1-4281-9d5d-cdab5506a864" (UID: "4e913a61-bea1-4281-9d5d-cdab5506a864"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:34:17.175270 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:17.175210    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e913a61-bea1-4281-9d5d-cdab5506a864-istio-token" pod "4e913a61-bea1-4281-9d5d-cdab5506a864" (UID: "4e913a61-bea1-4281-9d5d-cdab5506a864"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:34:17.274537 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:17.274501    5678 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e913a61-bea1-4281-9d5d-cdab5506a864-istio-token\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:17.274537 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:17.274530    5678 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e913a61-bea1-4281-9d5d-cdab5506a864-istio-podinfo\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:17.274537 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:17.274540    5678 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e913a61-bea1-4281-9d5d-cdab5506a864-istio-envoy\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:17.274811 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:17.274549    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hczps\" (UniqueName: \"kubernetes.io/projected/4e913a61-bea1-4281-9d5d-cdab5506a864-kube-api-access-hczps\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:17.383943 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:17.383909    5678 generic.go:356] "Generic (PLEG): container finished" podID="4e913a61-bea1-4281-9d5d-cdab5506a864" containerID="2f6d42261c7f0ce6f5c924ba6332ac7eb03692406eb0200a79b0b374ef17d304" exitCode=0
Oct 30 22:34:17.384118 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:17.383961    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cqvp9"
Oct 30 22:34:17.384118 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:17.383976    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cqvp9" event={"ID":"4e913a61-bea1-4281-9d5d-cdab5506a864","Type":"ContainerDied","Data":"2f6d42261c7f0ce6f5c924ba6332ac7eb03692406eb0200a79b0b374ef17d304"}
Oct 30 22:34:17.384118 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:17.384050    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cqvp9" event={"ID":"4e913a61-bea1-4281-9d5d-cdab5506a864","Type":"ContainerDied","Data":"2472b400321d9a70c466ba473f4c76c48057ea4e6c843f2f25bd74375c3174f1"}
Oct 30 22:34:17.384118 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:17.384073    5678 scope.go:122] "RemoveContainer" containerID="2f6d42261c7f0ce6f5c924ba6332ac7eb03692406eb0200a79b0b374ef17d304"
Oct 30 22:34:17.397113 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:17.397088    5678 scope.go:122] "RemoveContainer" containerID="2f6d42261c7f0ce6f5c924ba6332ac7eb03692406eb0200a79b0b374ef17d304"
Oct 30 22:34:17.397589 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:34:17.397513    5678 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f6d42261c7f0ce6f5c924ba6332ac7eb03692406eb0200a79b0b374ef17d304\": container with ID starting with 2f6d42261c7f0ce6f5c924ba6332ac7eb03692406eb0200a79b0b374ef17d304 not found: ID does not exist" containerID="2f6d42261c7f0ce6f5c924ba6332ac7eb03692406eb0200a79b0b374ef17d304"
Oct 30 22:34:17.397589 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:17.397552    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f6d42261c7f0ce6f5c924ba6332ac7eb03692406eb0200a79b0b374ef17d304"} err="failed to get container status \"2f6d42261c7f0ce6f5c924ba6332ac7eb03692406eb0200a79b0b374ef17d304\": rpc error: code = NotFound desc = could not find container \"2f6d42261c7f0ce6f5c924ba6332ac7eb03692406eb0200a79b0b374ef17d304\": container with ID starting with 2f6d42261c7f0ce6f5c924ba6332ac7eb03692406eb0200a79b0b374ef17d304 not found: ID does not exist"
Oct 30 22:34:17.544896 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:17.544769    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cqvp9"]
Oct 30 22:34:17.551851 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:17.551820    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-cqvp9"]
Oct 30 22:34:18.395311 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:18.395104    5678 generic.go:356] "Generic (PLEG): container finished" podID="e400e450-98f2-423b-bbc1-478fb88f61a2" containerID="fdeb0d71791c9066e532453a2f5662b6bd9cea6cdfea15e7c29496a2e09ca469" exitCode=0
Oct 30 22:34:18.395311 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:18.395179    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-qp9q4" event={"ID":"e400e450-98f2-423b-bbc1-478fb88f61a2","Type":"ContainerDied","Data":"fdeb0d71791c9066e532453a2f5662b6bd9cea6cdfea15e7c29496a2e09ca469"}
Oct 30 22:34:18.395311 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:18.395232    5678 scope.go:122] "RemoveContainer" containerID="6b480b2933e1c4531fb1444aed1d873b1593d1a10905c1b7420b8291ce7ada3c"
Oct 30 22:34:18.426801 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:18.426777    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-qp9q4"
Oct 30 22:34:18.487226 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:18.487196    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e400e450-98f2-423b-bbc1-478fb88f61a2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e400e450-98f2-423b-bbc1-478fb88f61a2-workload-socket\") pod \"e400e450-98f2-423b-bbc1-478fb88f61a2\" (UID: \"e400e450-98f2-423b-bbc1-478fb88f61a2\") "
Oct 30 22:34:18.487367 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:18.487235    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e400e450-98f2-423b-bbc1-478fb88f61a2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e400e450-98f2-423b-bbc1-478fb88f61a2-istio-podinfo\") pod \"e400e450-98f2-423b-bbc1-478fb88f61a2\" (UID: \"e400e450-98f2-423b-bbc1-478fb88f61a2\") "
Oct 30 22:34:18.487367 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:18.487333    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e400e450-98f2-423b-bbc1-478fb88f61a2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e400e450-98f2-423b-bbc1-478fb88f61a2-istio-envoy\") pod \"e400e450-98f2-423b-bbc1-478fb88f61a2\" (UID: \"e400e450-98f2-423b-bbc1-478fb88f61a2\") "
Oct 30 22:34:18.487502 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:18.487385    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e400e450-98f2-423b-bbc1-478fb88f61a2-istio-token\" (UniqueName: \"kubernetes.io/projected/e400e450-98f2-423b-bbc1-478fb88f61a2-istio-token\") pod \"e400e450-98f2-423b-bbc1-478fb88f61a2\" (UID: \"e400e450-98f2-423b-bbc1-478fb88f61a2\") "
Oct 30 22:34:18.487502 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:18.487431    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e400e450-98f2-423b-bbc1-478fb88f61a2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e400e450-98f2-423b-bbc1-478fb88f61a2-istiod-ca-cert\") pod \"e400e450-98f2-423b-bbc1-478fb88f61a2\" (UID: \"e400e450-98f2-423b-bbc1-478fb88f61a2\") "
Oct 30 22:34:18.487502 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:18.487469    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e400e450-98f2-423b-bbc1-478fb88f61a2-kube-api-access-xwn6c\" (UniqueName: \"kubernetes.io/projected/e400e450-98f2-423b-bbc1-478fb88f61a2-kube-api-access-xwn6c\") pod \"e400e450-98f2-423b-bbc1-478fb88f61a2\" (UID: \"e400e450-98f2-423b-bbc1-478fb88f61a2\") "
Oct 30 22:34:18.487502 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:18.487495    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e400e450-98f2-423b-bbc1-478fb88f61a2-workload-socket" pod "e400e450-98f2-423b-bbc1-478fb88f61a2" (UID: "e400e450-98f2-423b-bbc1-478fb88f61a2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:34:18.487730 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:18.487519    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e400e450-98f2-423b-bbc1-478fb88f61a2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e400e450-98f2-423b-bbc1-478fb88f61a2-istio-data\") pod \"e400e450-98f2-423b-bbc1-478fb88f61a2\" (UID: \"e400e450-98f2-423b-bbc1-478fb88f61a2\") "
Oct 30 22:34:18.487730 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:18.487579    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e400e450-98f2-423b-bbc1-478fb88f61a2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e400e450-98f2-423b-bbc1-478fb88f61a2-credential-socket\") pod \"e400e450-98f2-423b-bbc1-478fb88f61a2\" (UID: \"e400e450-98f2-423b-bbc1-478fb88f61a2\") "
Oct 30 22:34:18.487730 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:18.487684    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e400e450-98f2-423b-bbc1-478fb88f61a2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e400e450-98f2-423b-bbc1-478fb88f61a2-workload-certs\") pod \"e400e450-98f2-423b-bbc1-478fb88f61a2\" (UID: \"e400e450-98f2-423b-bbc1-478fb88f61a2\") "
Oct 30 22:34:18.487886 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:18.487762    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e400e450-98f2-423b-bbc1-478fb88f61a2-istio-data" pod "e400e450-98f2-423b-bbc1-478fb88f61a2" (UID: "e400e450-98f2-423b-bbc1-478fb88f61a2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:34:18.487886 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:18.487862    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e400e450-98f2-423b-bbc1-478fb88f61a2-credential-socket" pod "e400e450-98f2-423b-bbc1-478fb88f61a2" (UID: "e400e450-98f2-423b-bbc1-478fb88f61a2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:34:18.487886 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:18.487882    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e400e450-98f2-423b-bbc1-478fb88f61a2-istiod-ca-cert" pod "e400e450-98f2-423b-bbc1-478fb88f61a2" (UID: "e400e450-98f2-423b-bbc1-478fb88f61a2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:34:18.488078 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:18.488038    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e400e450-98f2-423b-bbc1-478fb88f61a2-workload-certs" pod "e400e450-98f2-423b-bbc1-478fb88f61a2" (UID: "e400e450-98f2-423b-bbc1-478fb88f61a2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:34:18.488202 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:18.488175    5678 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e400e450-98f2-423b-bbc1-478fb88f61a2-workload-socket\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:18.488341 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:18.488207    5678 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e400e450-98f2-423b-bbc1-478fb88f61a2-istiod-ca-cert\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:18.488341 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:18.488221    5678 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e400e450-98f2-423b-bbc1-478fb88f61a2-istio-data\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:18.488341 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:18.488236    5678 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e400e450-98f2-423b-bbc1-478fb88f61a2-credential-socket\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:18.488341 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:18.488247    5678 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e400e450-98f2-423b-bbc1-478fb88f61a2-workload-certs\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:18.489740 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:18.489710    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e400e450-98f2-423b-bbc1-478fb88f61a2-istio-token" pod "e400e450-98f2-423b-bbc1-478fb88f61a2" (UID: "e400e450-98f2-423b-bbc1-478fb88f61a2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:34:18.489847 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:18.489718    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e400e450-98f2-423b-bbc1-478fb88f61a2-kube-api-access-xwn6c" pod "e400e450-98f2-423b-bbc1-478fb88f61a2" (UID: "e400e450-98f2-423b-bbc1-478fb88f61a2"). InnerVolumeSpecName "kube-api-access-xwn6c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:34:18.489911 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:18.489881    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e400e450-98f2-423b-bbc1-478fb88f61a2-istio-envoy" pod "e400e450-98f2-423b-bbc1-478fb88f61a2" (UID: "e400e450-98f2-423b-bbc1-478fb88f61a2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:34:18.489978 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:18.489921    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e400e450-98f2-423b-bbc1-478fb88f61a2-istio-podinfo" pod "e400e450-98f2-423b-bbc1-478fb88f61a2" (UID: "e400e450-98f2-423b-bbc1-478fb88f61a2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:34:18.523157 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:18.523127    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e913a61-bea1-4281-9d5d-cdab5506a864" path="/var/lib/kubelet/pods/4e913a61-bea1-4281-9d5d-cdab5506a864/volumes"
Oct 30 22:34:18.589204 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:18.589161    5678 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e400e450-98f2-423b-bbc1-478fb88f61a2-istio-envoy\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:18.589204 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:18.589194    5678 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e400e450-98f2-423b-bbc1-478fb88f61a2-istio-token\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:18.589204 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:18.589210    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwn6c\" (UniqueName: \"kubernetes.io/projected/e400e450-98f2-423b-bbc1-478fb88f61a2-kube-api-access-xwn6c\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:18.589566 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:18.589224    5678 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e400e450-98f2-423b-bbc1-478fb88f61a2-istio-podinfo\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:18.677795 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:18.677715    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-582593/test-model-582593-kserve-858995b55-6p46x"]
Oct 30 22:34:18.678283 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:18.678270    5678 cpu_manager.go:414] "containerMap: removing container" podUID="e400e450-98f2-423b-bbc1-478fb88f61a2" containerName="istio-proxy"
Oct 30 22:34:18.678325 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:18.678284    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e400e450-98f2-423b-bbc1-478fb88f61a2" containerName="istio-proxy"
Oct 30 22:34:18.678325 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:18.678303    5678 cpu_manager.go:414] "containerMap: removing container" podUID="5f9fdabd-6c3b-4bbe-b1ea-ef2159076fc1" containerName="payload-processing"
Oct 30 22:34:18.678325 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:18.678308    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f9fdabd-6c3b-4bbe-b1ea-ef2159076fc1" containerName="payload-processing"
Oct 30 22:34:18.678325 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:18.678321    5678 cpu_manager.go:414] "containerMap: removing container" podUID="e400e450-98f2-423b-bbc1-478fb88f61a2" containerName="istio-proxy"
Oct 30 22:34:18.678472 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:18.678326    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e400e450-98f2-423b-bbc1-478fb88f61a2" containerName="istio-proxy"
Oct 30 22:34:18.678472 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:18.678336    5678 cpu_manager.go:414] "containerMap: removing container" podUID="4e913a61-bea1-4281-9d5d-cdab5506a864" containerName="istio-proxy"
Oct 30 22:34:18.678472 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:18.678341    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e913a61-bea1-4281-9d5d-cdab5506a864" containerName="istio-proxy"
Oct 30 22:34:18.678472 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:18.678352    5678 cpu_manager.go:414] "containerMap: removing container" podUID="6b45b1a6-24c2-4170-b914-85c2a9a5aab8" containerName="payload-pre-processing"
Oct 30 22:34:18.678472 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:18.678358    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b45b1a6-24c2-4170-b914-85c2a9a5aab8" containerName="payload-pre-processing"
Oct 30 22:34:18.678626 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:18.678473    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="e400e450-98f2-423b-bbc1-478fb88f61a2" containerName="istio-proxy"
Oct 30 22:34:18.678626 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:18.678485    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f9fdabd-6c3b-4bbe-b1ea-ef2159076fc1" containerName="payload-processing"
Oct 30 22:34:18.678626 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:18.678493    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="e400e450-98f2-423b-bbc1-478fb88f61a2" containerName="istio-proxy"
Oct 30 22:34:18.678626 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:18.678499    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e913a61-bea1-4281-9d5d-cdab5506a864" containerName="istio-proxy"
Oct 30 22:34:18.678626 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:18.678506    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b45b1a6-24c2-4170-b914-85c2a9a5aab8" containerName="payload-pre-processing"
Oct 30 22:34:18.683657 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:18.683635    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-582593/test-model-582593-kserve-858995b55-6p46x"
Oct 30 22:34:18.689762 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:18.689734    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jj5l\" (UniqueName: \"kubernetes.io/projected/c8157ccd-c3b7-4b4b-8dcb-9a8881eaff26-kube-api-access-5jj5l\") pod \"test-model-582593-kserve-858995b55-6p46x\" (UID: \"c8157ccd-c3b7-4b4b-8dcb-9a8881eaff26\") " pod="ai-tenant-e2e-inf-a-582593/test-model-582593-kserve-858995b55-6p46x"
Oct 30 22:34:18.689903 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:18.689784    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8157ccd-c3b7-4b4b-8dcb-9a8881eaff26-tmp-dir\") pod \"test-model-582593-kserve-858995b55-6p46x\" (UID: \"c8157ccd-c3b7-4b4b-8dcb-9a8881eaff26\") " pod="ai-tenant-e2e-inf-a-582593/test-model-582593-kserve-858995b55-6p46x"
Oct 30 22:34:18.689903 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:18.689883    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8157ccd-c3b7-4b4b-8dcb-9a8881eaff26-home\") pod \"test-model-582593-kserve-858995b55-6p46x\" (UID: \"c8157ccd-c3b7-4b4b-8dcb-9a8881eaff26\") " pod="ai-tenant-e2e-inf-a-582593/test-model-582593-kserve-858995b55-6p46x"
Oct 30 22:34:18.690022 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:18.689927    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8157ccd-c3b7-4b4b-8dcb-9a8881eaff26-model-cache\") pod \"test-model-582593-kserve-858995b55-6p46x\" (UID: \"c8157ccd-c3b7-4b4b-8dcb-9a8881eaff26\") " pod="ai-tenant-e2e-inf-a-582593/test-model-582593-kserve-858995b55-6p46x"
Oct 30 22:34:18.690022 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:18.690014    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8157ccd-c3b7-4b4b-8dcb-9a8881eaff26-dshm\") pod \"test-model-582593-kserve-858995b55-6p46x\" (UID: \"c8157ccd-c3b7-4b4b-8dcb-9a8881eaff26\") " pod="ai-tenant-e2e-inf-a-582593/test-model-582593-kserve-858995b55-6p46x"
Oct 30 22:34:18.690169 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:18.690149    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8157ccd-c3b7-4b4b-8dcb-9a8881eaff26-tls-certs\") pod \"test-model-582593-kserve-858995b55-6p46x\" (UID: \"c8157ccd-c3b7-4b4b-8dcb-9a8881eaff26\") " pod="ai-tenant-e2e-inf-a-582593/test-model-582593-kserve-858995b55-6p46x"
Oct 30 22:34:18.735872 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:18.735842    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-582593\"/\"openshift-service-ca.crt\""
Oct 30 22:34:18.736078 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:18.735918    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-582593\"/\"default-dockercfg-xz5w2\""
Oct 30 22:34:18.736078 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:18.735927    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-582593\"/\"test-model-582593-kserve-self-signed-certs\""
Oct 30 22:34:18.736202 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:18.736105    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-582593\"/\"kube-root-ca.crt\""
Oct 30 22:34:18.738962 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:18.738938    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-582593/test-model-582593-kserve-858995b55-6p46x"]
Oct 30 22:34:18.790908 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:18.790883    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8157ccd-c3b7-4b4b-8dcb-9a8881eaff26-dshm\") pod \"test-model-582593-kserve-858995b55-6p46x\" (UID: \"c8157ccd-c3b7-4b4b-8dcb-9a8881eaff26\") " pod="ai-tenant-e2e-inf-a-582593/test-model-582593-kserve-858995b55-6p46x"
Oct 30 22:34:18.791037 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:18.790943    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8157ccd-c3b7-4b4b-8dcb-9a8881eaff26-tls-certs\") pod \"test-model-582593-kserve-858995b55-6p46x\" (UID: \"c8157ccd-c3b7-4b4b-8dcb-9a8881eaff26\") " pod="ai-tenant-e2e-inf-a-582593/test-model-582593-kserve-858995b55-6p46x"
Oct 30 22:34:18.791037 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:18.790971    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jj5l\" (UniqueName: \"kubernetes.io/projected/c8157ccd-c3b7-4b4b-8dcb-9a8881eaff26-kube-api-access-5jj5l\") pod \"test-model-582593-kserve-858995b55-6p46x\" (UID: \"c8157ccd-c3b7-4b4b-8dcb-9a8881eaff26\") " pod="ai-tenant-e2e-inf-a-582593/test-model-582593-kserve-858995b55-6p46x"
Oct 30 22:34:18.791037 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:18.790999    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8157ccd-c3b7-4b4b-8dcb-9a8881eaff26-tmp-dir\") pod \"test-model-582593-kserve-858995b55-6p46x\" (UID: \"c8157ccd-c3b7-4b4b-8dcb-9a8881eaff26\") " pod="ai-tenant-e2e-inf-a-582593/test-model-582593-kserve-858995b55-6p46x"
Oct 30 22:34:18.791189 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:18.791039    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8157ccd-c3b7-4b4b-8dcb-9a8881eaff26-home\") pod \"test-model-582593-kserve-858995b55-6p46x\" (UID: \"c8157ccd-c3b7-4b4b-8dcb-9a8881eaff26\") " pod="ai-tenant-e2e-inf-a-582593/test-model-582593-kserve-858995b55-6p46x"
Oct 30 22:34:18.791189 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:18.791065    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8157ccd-c3b7-4b4b-8dcb-9a8881eaff26-model-cache\") pod \"test-model-582593-kserve-858995b55-6p46x\" (UID: \"c8157ccd-c3b7-4b4b-8dcb-9a8881eaff26\") " pod="ai-tenant-e2e-inf-a-582593/test-model-582593-kserve-858995b55-6p46x"
Oct 30 22:34:18.791462 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:18.791439    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8157ccd-c3b7-4b4b-8dcb-9a8881eaff26-home\") pod \"test-model-582593-kserve-858995b55-6p46x\" (UID: \"c8157ccd-c3b7-4b4b-8dcb-9a8881eaff26\") " pod="ai-tenant-e2e-inf-a-582593/test-model-582593-kserve-858995b55-6p46x"
Oct 30 22:34:18.791579 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:18.791438    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8157ccd-c3b7-4b4b-8dcb-9a8881eaff26-tmp-dir\") pod \"test-model-582593-kserve-858995b55-6p46x\" (UID: \"c8157ccd-c3b7-4b4b-8dcb-9a8881eaff26\") " pod="ai-tenant-e2e-inf-a-582593/test-model-582593-kserve-858995b55-6p46x"
Oct 30 22:34:18.791579 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:18.791487    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8157ccd-c3b7-4b4b-8dcb-9a8881eaff26-model-cache\") pod \"test-model-582593-kserve-858995b55-6p46x\" (UID: \"c8157ccd-c3b7-4b4b-8dcb-9a8881eaff26\") " pod="ai-tenant-e2e-inf-a-582593/test-model-582593-kserve-858995b55-6p46x"
Oct 30 22:34:18.793723 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:18.793702    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8157ccd-c3b7-4b4b-8dcb-9a8881eaff26-dshm\") pod \"test-model-582593-kserve-858995b55-6p46x\" (UID: \"c8157ccd-c3b7-4b4b-8dcb-9a8881eaff26\") " pod="ai-tenant-e2e-inf-a-582593/test-model-582593-kserve-858995b55-6p46x"
Oct 30 22:34:18.794006 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:18.793989    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8157ccd-c3b7-4b4b-8dcb-9a8881eaff26-tls-certs\") pod \"test-model-582593-kserve-858995b55-6p46x\" (UID: \"c8157ccd-c3b7-4b4b-8dcb-9a8881eaff26\") " pod="ai-tenant-e2e-inf-a-582593/test-model-582593-kserve-858995b55-6p46x"
Oct 30 22:34:18.851480 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:18.851451    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jj5l\" (UniqueName: \"kubernetes.io/projected/c8157ccd-c3b7-4b4b-8dcb-9a8881eaff26-kube-api-access-5jj5l\") pod \"test-model-582593-kserve-858995b55-6p46x\" (UID: \"c8157ccd-c3b7-4b4b-8dcb-9a8881eaff26\") " pod="ai-tenant-e2e-inf-a-582593/test-model-582593-kserve-858995b55-6p46x"
Oct 30 22:34:18.996258 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:18.996177    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-582593/test-model-582593-kserve-858995b55-6p46x"
Oct 30 22:34:19.277454 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:19.277380    5678 patch_prober.go:29] interesting pod/e2e-worker-w2-bc714d-openshift-default-67558b95cf-wqqzd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:34:19.277587 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:19.277468    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-wqqzd" podUID="99f99481-73d0-4fb4-9c88-12ee143a6904" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:34:19.386779 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:19.386748    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-582593/test-model-582593-kserve-858995b55-6p46x"]
Oct 30 22:34:19.393652 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:34:19.393627    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc8157ccd_c3b7_4b4b_8dcb_9a8881eaff26.slice/crio-4fa25500cfa81e535f69b246b1000520459bbcd9160d33f78d90afc831058cb7 WatchSource:0}: Error finding container 4fa25500cfa81e535f69b246b1000520459bbcd9160d33f78d90afc831058cb7: Status 404 returned error can't find the container with id 4fa25500cfa81e535f69b246b1000520459bbcd9160d33f78d90afc831058cb7
Oct 30 22:34:19.402611 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:19.402582    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-582593/test-model-582593-kserve-858995b55-6p46x" event={"ID":"c8157ccd-c3b7-4b4b-8dcb-9a8881eaff26","Type":"ContainerStarted","Data":"4fa25500cfa81e535f69b246b1000520459bbcd9160d33f78d90afc831058cb7"}
Oct 30 22:34:19.404185 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:19.404164    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-qp9q4" event={"ID":"e400e450-98f2-423b-bbc1-478fb88f61a2","Type":"ContainerDied","Data":"c7cfb1978eca7aea1fd6da84e51952f9c2afd520ea1fde298e0e176a0296e0ac"}
Oct 30 22:34:19.404292 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:19.404181    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-qp9q4"
Oct 30 22:34:19.404292 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:19.404209    5678 scope.go:122] "RemoveContainer" containerID="fdeb0d71791c9066e532453a2f5662b6bd9cea6cdfea15e7c29496a2e09ca469"
Oct 30 22:34:19.628453 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:19.628424    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-qp9q4"]
Oct 30 22:34:19.633595 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:19.633567    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-qp9q4"]
Oct 30 22:34:20.420836 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:20.420792    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-582593/test-model-582593-kserve-858995b55-6p46x" event={"ID":"c8157ccd-c3b7-4b4b-8dcb-9a8881eaff26","Type":"ContainerStarted","Data":"69bf713b7aeaaa5ad79eecabdc6dc749006b4cfb86171ca0ce76103b3ffb5bf5"}
Oct 30 22:34:20.423734 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:20.423706    5678 generic.go:356] "Generic (PLEG): container finished" podID="99f99481-73d0-4fb4-9c88-12ee143a6904" containerID="ba02e73767820b8f4d9f9f6c922d0537157b67d8705891fb420aabb534e608c4" exitCode=0
Oct 30 22:34:20.423885 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:20.423779    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-wqqzd" event={"ID":"99f99481-73d0-4fb4-9c88-12ee143a6904","Type":"ContainerDied","Data":"ba02e73767820b8f4d9f9f6c922d0537157b67d8705891fb420aabb534e608c4"}
Oct 30 22:34:20.423885 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:20.423879    5678 scope.go:122] "RemoveContainer" containerID="1850898ccd8dc9a1d5251132a95b3f1e312f10352d5a5baaea7fc97ccbe83434"
Oct 30 22:34:20.525046 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:20.525009    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e400e450-98f2-423b-bbc1-478fb88f61a2" path="/var/lib/kubelet/pods/e400e450-98f2-423b-bbc1-478fb88f61a2/volumes"
Oct 30 22:34:20.549292 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:20.549248    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-582593/test-model-582593-kserve-858995b55-6p46x" podStartSLOduration=2.549235499 podStartE2EDuration="2.549235499s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:34:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:34:20.548608816 +0000 UTC m=+2638.152916370" watchObservedRunningTime="2026-10-30 22:34:20.549235499 +0000 UTC m=+2638.153543051"
Oct 30 22:34:20.558824 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:20.558773    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-wqqzd"
Oct 30 22:34:20.611676 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:20.611651    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99f99481-73d0-4fb4-9c88-12ee143a6904-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99f99481-73d0-4fb4-9c88-12ee143a6904-credential-socket\") pod \"99f99481-73d0-4fb4-9c88-12ee143a6904\" (UID: \"99f99481-73d0-4fb4-9c88-12ee143a6904\") "
Oct 30 22:34:20.611824 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:20.611727    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/99f99481-73d0-4fb4-9c88-12ee143a6904-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99f99481-73d0-4fb4-9c88-12ee143a6904-istio-podinfo\") pod \"99f99481-73d0-4fb4-9c88-12ee143a6904\" (UID: \"99f99481-73d0-4fb4-9c88-12ee143a6904\") "
Oct 30 22:34:20.611824 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:20.611753    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99f99481-73d0-4fb4-9c88-12ee143a6904-kube-api-access-s95sf\" (UniqueName: \"kubernetes.io/projected/99f99481-73d0-4fb4-9c88-12ee143a6904-kube-api-access-s95sf\") pod \"99f99481-73d0-4fb4-9c88-12ee143a6904\" (UID: \"99f99481-73d0-4fb4-9c88-12ee143a6904\") "
Oct 30 22:34:20.611824 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:20.611788    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99f99481-73d0-4fb4-9c88-12ee143a6904-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99f99481-73d0-4fb4-9c88-12ee143a6904-istiod-ca-cert\") pod \"99f99481-73d0-4fb4-9c88-12ee143a6904\" (UID: \"99f99481-73d0-4fb4-9c88-12ee143a6904\") "
Oct 30 22:34:20.611824 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:20.611812    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99f99481-73d0-4fb4-9c88-12ee143a6904-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99f99481-73d0-4fb4-9c88-12ee143a6904-istio-envoy\") pod \"99f99481-73d0-4fb4-9c88-12ee143a6904\" (UID: \"99f99481-73d0-4fb4-9c88-12ee143a6904\") "
Oct 30 22:34:20.612038 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:20.611927    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99f99481-73d0-4fb4-9c88-12ee143a6904-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99f99481-73d0-4fb4-9c88-12ee143a6904-istio-data\") pod \"99f99481-73d0-4fb4-9c88-12ee143a6904\" (UID: \"99f99481-73d0-4fb4-9c88-12ee143a6904\") "
Oct 30 22:34:20.612038 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:20.611923    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99f99481-73d0-4fb4-9c88-12ee143a6904-credential-socket" pod "99f99481-73d0-4fb4-9c88-12ee143a6904" (UID: "99f99481-73d0-4fb4-9c88-12ee143a6904"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:34:20.612152 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:20.612005    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99f99481-73d0-4fb4-9c88-12ee143a6904-istio-token\" (UniqueName: \"kubernetes.io/projected/99f99481-73d0-4fb4-9c88-12ee143a6904-istio-token\") pod \"99f99481-73d0-4fb4-9c88-12ee143a6904\" (UID: \"99f99481-73d0-4fb4-9c88-12ee143a6904\") "
Oct 30 22:34:20.612152 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:20.612091    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99f99481-73d0-4fb4-9c88-12ee143a6904-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99f99481-73d0-4fb4-9c88-12ee143a6904-workload-certs\") pod \"99f99481-73d0-4fb4-9c88-12ee143a6904\" (UID: \"99f99481-73d0-4fb4-9c88-12ee143a6904\") "
Oct 30 22:34:20.612268 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:20.612167    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99f99481-73d0-4fb4-9c88-12ee143a6904-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99f99481-73d0-4fb4-9c88-12ee143a6904-workload-socket\") pod \"99f99481-73d0-4fb4-9c88-12ee143a6904\" (UID: \"99f99481-73d0-4fb4-9c88-12ee143a6904\") "
Oct 30 22:34:20.612268 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:20.612203    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99f99481-73d0-4fb4-9c88-12ee143a6904-istio-data" pod "99f99481-73d0-4fb4-9c88-12ee143a6904" (UID: "99f99481-73d0-4fb4-9c88-12ee143a6904"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:34:20.612268 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:20.612213    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99f99481-73d0-4fb4-9c88-12ee143a6904-istiod-ca-cert" pod "99f99481-73d0-4fb4-9c88-12ee143a6904" (UID: "99f99481-73d0-4fb4-9c88-12ee143a6904"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:34:20.612500 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:20.612478    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99f99481-73d0-4fb4-9c88-12ee143a6904-workload-certs" pod "99f99481-73d0-4fb4-9c88-12ee143a6904" (UID: "99f99481-73d0-4fb4-9c88-12ee143a6904"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:34:20.612589 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:20.612513    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99f99481-73d0-4fb4-9c88-12ee143a6904-workload-socket" pod "99f99481-73d0-4fb4-9c88-12ee143a6904" (UID: "99f99481-73d0-4fb4-9c88-12ee143a6904"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:34:20.612790 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:20.612773    5678 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99f99481-73d0-4fb4-9c88-12ee143a6904-workload-socket\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:20.612881 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:20.612799    5678 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99f99481-73d0-4fb4-9c88-12ee143a6904-credential-socket\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:20.612881 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:20.612814    5678 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99f99481-73d0-4fb4-9c88-12ee143a6904-istiod-ca-cert\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:20.612881 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:20.612829    5678 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99f99481-73d0-4fb4-9c88-12ee143a6904-istio-data\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:20.612881 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:20.612842    5678 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99f99481-73d0-4fb4-9c88-12ee143a6904-workload-certs\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:20.614081 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:20.614059    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99f99481-73d0-4fb4-9c88-12ee143a6904-kube-api-access-s95sf" pod "99f99481-73d0-4fb4-9c88-12ee143a6904" (UID: "99f99481-73d0-4fb4-9c88-12ee143a6904"). InnerVolumeSpecName "kube-api-access-s95sf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:34:20.614253 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:20.614233    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99f99481-73d0-4fb4-9c88-12ee143a6904-istio-envoy" pod "99f99481-73d0-4fb4-9c88-12ee143a6904" (UID: "99f99481-73d0-4fb4-9c88-12ee143a6904"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:34:20.614483 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:20.614459    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99f99481-73d0-4fb4-9c88-12ee143a6904-istio-token" pod "99f99481-73d0-4fb4-9c88-12ee143a6904" (UID: "99f99481-73d0-4fb4-9c88-12ee143a6904"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:34:20.614575 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:20.614481    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/99f99481-73d0-4fb4-9c88-12ee143a6904-istio-podinfo" pod "99f99481-73d0-4fb4-9c88-12ee143a6904" (UID: "99f99481-73d0-4fb4-9c88-12ee143a6904"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:34:20.714298 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:20.714268    5678 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99f99481-73d0-4fb4-9c88-12ee143a6904-istio-envoy\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:20.714298 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:20.714297    5678 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99f99481-73d0-4fb4-9c88-12ee143a6904-istio-token\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:20.714544 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:20.714311    5678 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99f99481-73d0-4fb4-9c88-12ee143a6904-istio-podinfo\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:20.714544 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:20.714325    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s95sf\" (UniqueName: \"kubernetes.io/projected/99f99481-73d0-4fb4-9c88-12ee143a6904-kube-api-access-s95sf\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:20.996561 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:20.996527    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-582593/test-model-582593-kserve-858995b55-6p46x"
Oct 30 22:34:21.434980 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:21.434945    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-wqqzd"
Oct 30 22:34:21.434980 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:21.434967    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-wqqzd" event={"ID":"99f99481-73d0-4fb4-9c88-12ee143a6904","Type":"ContainerDied","Data":"f3df1686d80b7438a2cbbe728098857646e59cc5768b12b047bd47917b87389e"}
Oct 30 22:34:21.435745 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:21.435029    5678 scope.go:122] "RemoveContainer" containerID="ba02e73767820b8f4d9f9f6c922d0537157b67d8705891fb420aabb534e608c4"
Oct 30 22:34:21.593849 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:21.593815    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-wqqzd"]
Oct 30 22:34:21.640889 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:21.640854    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-wqqzd"]
Oct 30 22:34:21.930201 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:21.930168    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-779f32-66c9bd7459-g2qwk"]
Oct 30 22:34:22.524642 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:22.524610    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99f99481-73d0-4fb4-9c88-12ee143a6904" path="/var/lib/kubelet/pods/99f99481-73d0-4fb4-9c88-12ee143a6904/volumes"
Oct 30 22:34:26.930508 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:26.930461    5678 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-779f32-66c9bd7459-g2qwk" podUID="bdeffce0-5736-4ac8-89b1-cfdd262fd5c7" containerName="payload-processing" containerID="cri-o://4fbc28e64b4ec94e9f0c6ae2007d7f3b7541fbd8b23721a36dfdf378c87e576e" gracePeriod=25
Oct 30 22:34:28.996659 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:28.996630    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-582593/test-model-582593-kserve-858995b55-6p46x"
Oct 30 22:34:29.010126 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:29.010100    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-582593/test-model-582593-kserve-858995b55-6p46x"
Oct 30 22:34:29.495365 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:29.495335    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-582593/test-model-582593-kserve-858995b55-6p46x"
Oct 30 22:34:30.450424 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:30.450354    5678 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-779f32/premium-w0-779f32-kserve-67978797c8-jpkvl" podUID="40138fb9-9a75-4458-a351-11eb61c4a1e3" containerName="main" containerID="cri-o://f54bcac0d9e7ceb682de4e2c27e8d08fe11ac442a95ca35ad437b2acc431cac0" gracePeriod=45
Oct 30 22:34:30.701193 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:30.701133    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-779f32/premium-w0-779f32-kserve-67978797c8-jpkvl"
Oct 30 22:34:30.834278 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:30.834247    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40138fb9-9a75-4458-a351-11eb61c4a1e3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40138fb9-9a75-4458-a351-11eb61c4a1e3-model-cache\") pod \"40138fb9-9a75-4458-a351-11eb61c4a1e3\" (UID: \"40138fb9-9a75-4458-a351-11eb61c4a1e3\") "
Oct 30 22:34:30.834460 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:30.834379    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40138fb9-9a75-4458-a351-11eb61c4a1e3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/40138fb9-9a75-4458-a351-11eb61c4a1e3-dshm\") pod \"40138fb9-9a75-4458-a351-11eb61c4a1e3\" (UID: \"40138fb9-9a75-4458-a351-11eb61c4a1e3\") "
Oct 30 22:34:30.834460 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:30.834453    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40138fb9-9a75-4458-a351-11eb61c4a1e3-home\" (UniqueName: \"kubernetes.io/empty-dir/40138fb9-9a75-4458-a351-11eb61c4a1e3-home\") pod \"40138fb9-9a75-4458-a351-11eb61c4a1e3\" (UID: \"40138fb9-9a75-4458-a351-11eb61c4a1e3\") "
Oct 30 22:34:30.834598 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:30.834481    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40138fb9-9a75-4458-a351-11eb61c4a1e3-tls-certs\" (UniqueName: \"kubernetes.io/secret/40138fb9-9a75-4458-a351-11eb61c4a1e3-tls-certs\") pod \"40138fb9-9a75-4458-a351-11eb61c4a1e3\" (UID: \"40138fb9-9a75-4458-a351-11eb61c4a1e3\") "
Oct 30 22:34:30.834598 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:30.834526    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40138fb9-9a75-4458-a351-11eb61c4a1e3-model-cache" pod "40138fb9-9a75-4458-a351-11eb61c4a1e3" (UID: "40138fb9-9a75-4458-a351-11eb61c4a1e3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:34:30.834598 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:30.834533    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40138fb9-9a75-4458-a351-11eb61c4a1e3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40138fb9-9a75-4458-a351-11eb61c4a1e3-tmp-dir\") pod \"40138fb9-9a75-4458-a351-11eb61c4a1e3\" (UID: \"40138fb9-9a75-4458-a351-11eb61c4a1e3\") "
Oct 30 22:34:30.834753 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:30.834598    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40138fb9-9a75-4458-a351-11eb61c4a1e3-kube-api-access-mhrlr\" (UniqueName: \"kubernetes.io/projected/40138fb9-9a75-4458-a351-11eb61c4a1e3-kube-api-access-mhrlr\") pod \"40138fb9-9a75-4458-a351-11eb61c4a1e3\" (UID: \"40138fb9-9a75-4458-a351-11eb61c4a1e3\") "
Oct 30 22:34:30.834753 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:30.834738    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40138fb9-9a75-4458-a351-11eb61c4a1e3-tmp-dir" pod "40138fb9-9a75-4458-a351-11eb61c4a1e3" (UID: "40138fb9-9a75-4458-a351-11eb61c4a1e3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:34:30.834840 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:30.834754    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40138fb9-9a75-4458-a351-11eb61c4a1e3-home" pod "40138fb9-9a75-4458-a351-11eb61c4a1e3" (UID: "40138fb9-9a75-4458-a351-11eb61c4a1e3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:34:30.835093 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:30.835068    5678 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40138fb9-9a75-4458-a351-11eb61c4a1e3-home\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:30.835093 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:30.835092    5678 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40138fb9-9a75-4458-a351-11eb61c4a1e3-tmp-dir\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:30.835323 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:30.835102    5678 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40138fb9-9a75-4458-a351-11eb61c4a1e3-model-cache\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:30.836645 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:30.836618    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40138fb9-9a75-4458-a351-11eb61c4a1e3-tls-certs" pod "40138fb9-9a75-4458-a351-11eb61c4a1e3" (UID: "40138fb9-9a75-4458-a351-11eb61c4a1e3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:34:30.836762 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:30.836684    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40138fb9-9a75-4458-a351-11eb61c4a1e3-dshm" pod "40138fb9-9a75-4458-a351-11eb61c4a1e3" (UID: "40138fb9-9a75-4458-a351-11eb61c4a1e3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:34:30.836762 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:30.836748    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40138fb9-9a75-4458-a351-11eb61c4a1e3-kube-api-access-mhrlr" pod "40138fb9-9a75-4458-a351-11eb61c4a1e3" (UID: "40138fb9-9a75-4458-a351-11eb61c4a1e3"). InnerVolumeSpecName "kube-api-access-mhrlr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:34:30.936862 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:30.936827    5678 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40138fb9-9a75-4458-a351-11eb61c4a1e3-dshm\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:30.936862 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:30.936856    5678 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40138fb9-9a75-4458-a351-11eb61c4a1e3-tls-certs\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:30.936862 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:30.936865    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhrlr\" (UniqueName: \"kubernetes.io/projected/40138fb9-9a75-4458-a351-11eb61c4a1e3-kube-api-access-mhrlr\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:31.505191 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:31.505147    5678 generic.go:356] "Generic (PLEG): container finished" podID="40138fb9-9a75-4458-a351-11eb61c4a1e3" containerID="f54bcac0d9e7ceb682de4e2c27e8d08fe11ac442a95ca35ad437b2acc431cac0" exitCode=0
Oct 30 22:34:31.506073 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:31.505211    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-779f32/premium-w0-779f32-kserve-67978797c8-jpkvl" event={"ID":"40138fb9-9a75-4458-a351-11eb61c4a1e3","Type":"ContainerDied","Data":"f54bcac0d9e7ceb682de4e2c27e8d08fe11ac442a95ca35ad437b2acc431cac0"}
Oct 30 22:34:31.506073 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:31.505287    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-779f32/premium-w0-779f32-kserve-67978797c8-jpkvl" event={"ID":"40138fb9-9a75-4458-a351-11eb61c4a1e3","Type":"ContainerDied","Data":"555c56d6f5ba3296e5487c55eb769cadb6a650dda1607fe4fb6de0bcb45190e7"}
Oct 30 22:34:31.506073 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:31.505309    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-779f32/premium-w0-779f32-kserve-67978797c8-jpkvl"
Oct 30 22:34:31.506073 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:31.505312    5678 scope.go:122] "RemoveContainer" containerID="f54bcac0d9e7ceb682de4e2c27e8d08fe11ac442a95ca35ad437b2acc431cac0"
Oct 30 22:34:31.516780 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:31.516762    5678 scope.go:122] "RemoveContainer" containerID="f54bcac0d9e7ceb682de4e2c27e8d08fe11ac442a95ca35ad437b2acc431cac0"
Oct 30 22:34:31.517079 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:34:31.517057    5678 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f54bcac0d9e7ceb682de4e2c27e8d08fe11ac442a95ca35ad437b2acc431cac0\": container with ID starting with f54bcac0d9e7ceb682de4e2c27e8d08fe11ac442a95ca35ad437b2acc431cac0 not found: ID does not exist" containerID="f54bcac0d9e7ceb682de4e2c27e8d08fe11ac442a95ca35ad437b2acc431cac0"
Oct 30 22:34:31.517130 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:31.517093    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f54bcac0d9e7ceb682de4e2c27e8d08fe11ac442a95ca35ad437b2acc431cac0"} err="failed to get container status \"f54bcac0d9e7ceb682de4e2c27e8d08fe11ac442a95ca35ad437b2acc431cac0\": rpc error: code = NotFound desc = could not find container \"f54bcac0d9e7ceb682de4e2c27e8d08fe11ac442a95ca35ad437b2acc431cac0\": container with ID starting with f54bcac0d9e7ceb682de4e2c27e8d08fe11ac442a95ca35ad437b2acc431cac0 not found: ID does not exist"
Oct 30 22:34:31.662227 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:31.662194    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-779f32/premium-w0-779f32-kserve-67978797c8-jpkvl"]
Oct 30 22:34:31.709316 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:31.709286    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-779f32/premium-w0-779f32-kserve-67978797c8-jpkvl"]
Oct 30 22:34:32.067678 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:32.067642    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-779f32-66c9bd7459-g2qwk" podUID="bdeffce0-5736-4ac8-89b1-cfdd262fd5c7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:34:32.522040 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:32.522010    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40138fb9-9a75-4458-a351-11eb61c4a1e3" path="/var/lib/kubelet/pods/40138fb9-9a75-4458-a351-11eb61c4a1e3/volumes"
Oct 30 22:34:40.571997 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:34:40.571964    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 22:34:42.067968 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:42.067928    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-779f32-66c9bd7459-g2qwk" podUID="bdeffce0-5736-4ac8-89b1-cfdd262fd5c7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:34:47.221259 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:47.221233    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-779f32-66c9bd7459-g2qwk"
Oct 30 22:34:47.319810 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:47.319781    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdeffce0-5736-4ac8-89b1-cfdd262fd5c7-kube-api-access-tn7pf\" (UniqueName: \"kubernetes.io/projected/bdeffce0-5736-4ac8-89b1-cfdd262fd5c7-kube-api-access-tn7pf\") pod \"bdeffce0-5736-4ac8-89b1-cfdd262fd5c7\" (UID: \"bdeffce0-5736-4ac8-89b1-cfdd262fd5c7\") "
Oct 30 22:34:47.319974 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:47.319896    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bdeffce0-5736-4ac8-89b1-cfdd262fd5c7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdeffce0-5736-4ac8-89b1-cfdd262fd5c7-plugins-config-volume\") pod \"bdeffce0-5736-4ac8-89b1-cfdd262fd5c7\" (UID: \"bdeffce0-5736-4ac8-89b1-cfdd262fd5c7\") "
Oct 30 22:34:47.320260 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:47.320238    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdeffce0-5736-4ac8-89b1-cfdd262fd5c7-plugins-config-volume" pod "bdeffce0-5736-4ac8-89b1-cfdd262fd5c7" (UID: "bdeffce0-5736-4ac8-89b1-cfdd262fd5c7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:34:47.320360 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:47.320345    5678 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdeffce0-5736-4ac8-89b1-cfdd262fd5c7-plugins-config-volume\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:47.321911 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:47.321890    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdeffce0-5736-4ac8-89b1-cfdd262fd5c7-kube-api-access-tn7pf" pod "bdeffce0-5736-4ac8-89b1-cfdd262fd5c7" (UID: "bdeffce0-5736-4ac8-89b1-cfdd262fd5c7"). InnerVolumeSpecName "kube-api-access-tn7pf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:34:47.421276 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:47.421241    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tn7pf\" (UniqueName: \"kubernetes.io/projected/bdeffce0-5736-4ac8-89b1-cfdd262fd5c7-kube-api-access-tn7pf\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:47.611235 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:47.611144    5678 generic.go:356] "Generic (PLEG): container finished" podID="bdeffce0-5736-4ac8-89b1-cfdd262fd5c7" containerID="4fbc28e64b4ec94e9f0c6ae2007d7f3b7541fbd8b23721a36dfdf378c87e576e" exitCode=0
Oct 30 22:34:47.611235 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:47.611197    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-779f32-66c9bd7459-g2qwk"
Oct 30 22:34:47.611235 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:47.611225    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-779f32-66c9bd7459-g2qwk" event={"ID":"bdeffce0-5736-4ac8-89b1-cfdd262fd5c7","Type":"ContainerDied","Data":"4fbc28e64b4ec94e9f0c6ae2007d7f3b7541fbd8b23721a36dfdf378c87e576e"}
Oct 30 22:34:47.611510 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:47.611273    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-779f32-66c9bd7459-g2qwk" event={"ID":"bdeffce0-5736-4ac8-89b1-cfdd262fd5c7","Type":"ContainerDied","Data":"9912a36c3edd9d447daff888c071264964a6f8532ef6e25b90a3096fcfaeddd1"}
Oct 30 22:34:47.611510 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:47.611294    5678 scope.go:122] "RemoveContainer" containerID="4fbc28e64b4ec94e9f0c6ae2007d7f3b7541fbd8b23721a36dfdf378c87e576e"
Oct 30 22:34:47.621102 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:47.621083    5678 scope.go:122] "RemoveContainer" containerID="4fbc28e64b4ec94e9f0c6ae2007d7f3b7541fbd8b23721a36dfdf378c87e576e"
Oct 30 22:34:47.621342 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:34:47.621325    5678 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fbc28e64b4ec94e9f0c6ae2007d7f3b7541fbd8b23721a36dfdf378c87e576e\": container with ID starting with 4fbc28e64b4ec94e9f0c6ae2007d7f3b7541fbd8b23721a36dfdf378c87e576e not found: ID does not exist" containerID="4fbc28e64b4ec94e9f0c6ae2007d7f3b7541fbd8b23721a36dfdf378c87e576e"
Oct 30 22:34:47.621392 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:47.621350    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fbc28e64b4ec94e9f0c6ae2007d7f3b7541fbd8b23721a36dfdf378c87e576e"} err="failed to get container status \"4fbc28e64b4ec94e9f0c6ae2007d7f3b7541fbd8b23721a36dfdf378c87e576e\": rpc error: code = NotFound desc = could not find container \"4fbc28e64b4ec94e9f0c6ae2007d7f3b7541fbd8b23721a36dfdf378c87e576e\": container with ID starting with 4fbc28e64b4ec94e9f0c6ae2007d7f3b7541fbd8b23721a36dfdf378c87e576e not found: ID does not exist"
Oct 30 22:34:47.780738 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:47.780704    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-779f32-66c9bd7459-g2qwk"]
Oct 30 22:34:47.785800 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:47.785763    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-779f32-66c9bd7459-g2qwk"]
Oct 30 22:34:48.522778 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:34:48.522745    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bdeffce0-5736-4ac8-89b1-cfdd262fd5c7" path="/var/lib/kubelet/pods/bdeffce0-5736-4ac8-89b1-cfdd262fd5c7/volumes"
Oct 30 22:35:10.571280 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:35:10.571243    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 22:35:40.571575 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:35:40.571498    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 22:35:54.866290 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:35:54.866256    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-t6pm5"]
Oct 30 22:35:54.866787 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:35:54.866533    5678 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-t6pm5" podUID="5d40decd-0eb0-4636-a8fe-06bbafd0484e" containerName="istio-proxy" containerID="cri-o://51e46e5c32fdd4104f6d5d325844c9f3a4051be47038c10876995bee67280709" gracePeriod=30
Oct 30 22:36:00.068472 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:36:00.068443    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-t6pm5" event={"ID":"5d40decd-0eb0-4636-a8fe-06bbafd0484e","Type":"ContainerDied","Data":"51e46e5c32fdd4104f6d5d325844c9f3a4051be47038c10876995bee67280709"}
Oct 30 22:36:00.068863 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:36:00.068434    5678 generic.go:356] "Generic (PLEG): container finished" podID="5d40decd-0eb0-4636-a8fe-06bbafd0484e" containerID="51e46e5c32fdd4104f6d5d325844c9f3a4051be47038c10876995bee67280709" exitCode=0
Oct 30 22:36:00.130831 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:36:00.130803    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-t6pm5"
Oct 30 22:36:00.226359 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:36:00.226273    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d40decd-0eb0-4636-a8fe-06bbafd0484e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d40decd-0eb0-4636-a8fe-06bbafd0484e-istiod-ca-cert\") pod \"5d40decd-0eb0-4636-a8fe-06bbafd0484e\" (UID: \"5d40decd-0eb0-4636-a8fe-06bbafd0484e\") "
Oct 30 22:36:00.226359 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:36:00.226314    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d40decd-0eb0-4636-a8fe-06bbafd0484e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d40decd-0eb0-4636-a8fe-06bbafd0484e-istio-envoy\") pod \"5d40decd-0eb0-4636-a8fe-06bbafd0484e\" (UID: \"5d40decd-0eb0-4636-a8fe-06bbafd0484e\") "
Oct 30 22:36:00.226359 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:36:00.226348    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d40decd-0eb0-4636-a8fe-06bbafd0484e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d40decd-0eb0-4636-a8fe-06bbafd0484e-istio-data\") pod \"5d40decd-0eb0-4636-a8fe-06bbafd0484e\" (UID: \"5d40decd-0eb0-4636-a8fe-06bbafd0484e\") "
Oct 30 22:36:00.226633 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:36:00.226449    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d40decd-0eb0-4636-a8fe-06bbafd0484e-kube-api-access-pvcpd\" (UniqueName: \"kubernetes.io/projected/5d40decd-0eb0-4636-a8fe-06bbafd0484e-kube-api-access-pvcpd\") pod \"5d40decd-0eb0-4636-a8fe-06bbafd0484e\" (UID: \"5d40decd-0eb0-4636-a8fe-06bbafd0484e\") "
Oct 30 22:36:00.226633 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:36:00.226491    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d40decd-0eb0-4636-a8fe-06bbafd0484e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d40decd-0eb0-4636-a8fe-06bbafd0484e-workload-socket\") pod \"5d40decd-0eb0-4636-a8fe-06bbafd0484e\" (UID: \"5d40decd-0eb0-4636-a8fe-06bbafd0484e\") "
Oct 30 22:36:00.226633 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:36:00.226571    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d40decd-0eb0-4636-a8fe-06bbafd0484e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d40decd-0eb0-4636-a8fe-06bbafd0484e-workload-certs\") pod \"5d40decd-0eb0-4636-a8fe-06bbafd0484e\" (UID: \"5d40decd-0eb0-4636-a8fe-06bbafd0484e\") "
Oct 30 22:36:00.226803 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:36:00.226630    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5d40decd-0eb0-4636-a8fe-06bbafd0484e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d40decd-0eb0-4636-a8fe-06bbafd0484e-istio-podinfo\") pod \"5d40decd-0eb0-4636-a8fe-06bbafd0484e\" (UID: \"5d40decd-0eb0-4636-a8fe-06bbafd0484e\") "
Oct 30 22:36:00.226803 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:36:00.226660    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d40decd-0eb0-4636-a8fe-06bbafd0484e-istio-token\" (UniqueName: \"kubernetes.io/projected/5d40decd-0eb0-4636-a8fe-06bbafd0484e-istio-token\") pod \"5d40decd-0eb0-4636-a8fe-06bbafd0484e\" (UID: \"5d40decd-0eb0-4636-a8fe-06bbafd0484e\") "
Oct 30 22:36:00.226803 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:36:00.226672    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d40decd-0eb0-4636-a8fe-06bbafd0484e-istio-data" pod "5d40decd-0eb0-4636-a8fe-06bbafd0484e" (UID: "5d40decd-0eb0-4636-a8fe-06bbafd0484e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:00.226803 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:36:00.226723    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d40decd-0eb0-4636-a8fe-06bbafd0484e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d40decd-0eb0-4636-a8fe-06bbafd0484e-credential-socket\") pod \"5d40decd-0eb0-4636-a8fe-06bbafd0484e\" (UID: \"5d40decd-0eb0-4636-a8fe-06bbafd0484e\") "
Oct 30 22:36:00.227017 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:36:00.226830    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d40decd-0eb0-4636-a8fe-06bbafd0484e-istiod-ca-cert" pod "5d40decd-0eb0-4636-a8fe-06bbafd0484e" (UID: "5d40decd-0eb0-4636-a8fe-06bbafd0484e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:36:00.227017 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:36:00.226872    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d40decd-0eb0-4636-a8fe-06bbafd0484e-workload-certs" pod "5d40decd-0eb0-4636-a8fe-06bbafd0484e" (UID: "5d40decd-0eb0-4636-a8fe-06bbafd0484e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:00.227115 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:36:00.227086    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d40decd-0eb0-4636-a8fe-06bbafd0484e-credential-socket" pod "5d40decd-0eb0-4636-a8fe-06bbafd0484e" (UID: "5d40decd-0eb0-4636-a8fe-06bbafd0484e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:00.227155 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:36:00.227104    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d40decd-0eb0-4636-a8fe-06bbafd0484e-workload-socket" pod "5d40decd-0eb0-4636-a8fe-06bbafd0484e" (UID: "5d40decd-0eb0-4636-a8fe-06bbafd0484e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:00.227338 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:36:00.227304    5678 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d40decd-0eb0-4636-a8fe-06bbafd0484e-istiod-ca-cert\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:00.227338 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:36:00.227331    5678 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d40decd-0eb0-4636-a8fe-06bbafd0484e-istio-data\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:00.227536 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:36:00.227348    5678 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d40decd-0eb0-4636-a8fe-06bbafd0484e-workload-socket\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:00.227536 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:36:00.227363    5678 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d40decd-0eb0-4636-a8fe-06bbafd0484e-workload-certs\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:00.227536 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:36:00.227378    5678 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d40decd-0eb0-4636-a8fe-06bbafd0484e-credential-socket\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:00.228762 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:36:00.228737    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5d40decd-0eb0-4636-a8fe-06bbafd0484e-istio-podinfo" pod "5d40decd-0eb0-4636-a8fe-06bbafd0484e" (UID: "5d40decd-0eb0-4636-a8fe-06bbafd0484e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:36:00.228884 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:36:00.228858    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d40decd-0eb0-4636-a8fe-06bbafd0484e-istio-envoy" pod "5d40decd-0eb0-4636-a8fe-06bbafd0484e" (UID: "5d40decd-0eb0-4636-a8fe-06bbafd0484e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:00.228947 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:36:00.228903    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d40decd-0eb0-4636-a8fe-06bbafd0484e-kube-api-access-pvcpd" pod "5d40decd-0eb0-4636-a8fe-06bbafd0484e" (UID: "5d40decd-0eb0-4636-a8fe-06bbafd0484e"). InnerVolumeSpecName "kube-api-access-pvcpd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:36:00.228947 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:36:00.228921    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d40decd-0eb0-4636-a8fe-06bbafd0484e-istio-token" pod "5d40decd-0eb0-4636-a8fe-06bbafd0484e" (UID: "5d40decd-0eb0-4636-a8fe-06bbafd0484e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:36:00.328711 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:36:00.328679    5678 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d40decd-0eb0-4636-a8fe-06bbafd0484e-istio-envoy\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:00.328711 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:36:00.328707    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvcpd\" (UniqueName: \"kubernetes.io/projected/5d40decd-0eb0-4636-a8fe-06bbafd0484e-kube-api-access-pvcpd\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:00.328711 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:36:00.328718    5678 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d40decd-0eb0-4636-a8fe-06bbafd0484e-istio-podinfo\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:00.329008 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:36:00.328726    5678 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d40decd-0eb0-4636-a8fe-06bbafd0484e-istio-token\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:01.075750 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:36:01.075722    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-t6pm5"
Oct 30 22:36:01.076192 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:36:01.075718    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-t6pm5" event={"ID":"5d40decd-0eb0-4636-a8fe-06bbafd0484e","Type":"ContainerDied","Data":"2e005984eacd3975b692661a5d974e32a5845c06a30417816e15c39988dffd53"}
Oct 30 22:36:01.076192 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:36:01.075857    5678 scope.go:122] "RemoveContainer" containerID="51e46e5c32fdd4104f6d5d325844c9f3a4051be47038c10876995bee67280709"
Oct 30 22:36:01.234018 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:36:01.233983    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-t6pm5"]
Oct 30 22:36:01.238798 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:36:01.238773    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-t6pm5"]
Oct 30 22:36:02.522987 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:36:02.522956    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d40decd-0eb0-4636-a8fe-06bbafd0484e" path="/var/lib/kubelet/pods/5d40decd-0eb0-4636-a8fe-06bbafd0484e/volumes"
Oct 30 22:36:10.571618 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:36:10.571581    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 22:36:40.571483 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:36:40.571442    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 22:37:10.572218 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:37:10.572129    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 22:37:34.408788 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:37:34.408755    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-582593/test-model-582593-kserve-858995b55-6p46x"]
Oct 30 22:37:40.572393 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:37:40.572351    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 22:37:49.428510 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:37:49.428457    5678 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-582593/test-model-582593-kserve-858995b55-6p46x" podUID="c8157ccd-c3b7-4b4b-8dcb-9a8881eaff26" containerName="main" containerID="cri-o://69bf713b7aeaaa5ad79eecabdc6dc749006b4cfb86171ca0ce76103b3ffb5bf5" gracePeriod=45
Oct 30 22:37:49.485589 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:37:49.485557    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-582593/test-model-582593-kserve-858995b55-6p46x" podUID="c8157ccd-c3b7-4b4b-8dcb-9a8881eaff26" containerName="main" probeResult="failure" output="Get \"https://10.134.0.166:8000/ready\": dial tcp 10.134.0.166:8000: connect: connection refused"
Oct 30 22:37:49.694087 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:37:49.694023    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-582593/test-model-582593-kserve-858995b55-6p46x"
Oct 30 22:37:49.729808 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:37:49.729779    5678 generic.go:356] "Generic (PLEG): container finished" podID="c8157ccd-c3b7-4b4b-8dcb-9a8881eaff26" containerID="69bf713b7aeaaa5ad79eecabdc6dc749006b4cfb86171ca0ce76103b3ffb5bf5" exitCode=0
Oct 30 22:37:49.729984 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:37:49.729820    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-582593/test-model-582593-kserve-858995b55-6p46x"
Oct 30 22:37:49.729984 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:37:49.729859    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-582593/test-model-582593-kserve-858995b55-6p46x" event={"ID":"c8157ccd-c3b7-4b4b-8dcb-9a8881eaff26","Type":"ContainerDied","Data":"69bf713b7aeaaa5ad79eecabdc6dc749006b4cfb86171ca0ce76103b3ffb5bf5"}
Oct 30 22:37:49.729984 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:37:49.729907    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-582593/test-model-582593-kserve-858995b55-6p46x" event={"ID":"c8157ccd-c3b7-4b4b-8dcb-9a8881eaff26","Type":"ContainerDied","Data":"4fa25500cfa81e535f69b246b1000520459bbcd9160d33f78d90afc831058cb7"}
Oct 30 22:37:49.729984 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:37:49.729928    5678 scope.go:122] "RemoveContainer" containerID="69bf713b7aeaaa5ad79eecabdc6dc749006b4cfb86171ca0ce76103b3ffb5bf5"
Oct 30 22:37:49.739224 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:37:49.739204    5678 scope.go:122] "RemoveContainer" containerID="69bf713b7aeaaa5ad79eecabdc6dc749006b4cfb86171ca0ce76103b3ffb5bf5"
Oct 30 22:37:49.739541 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:37:49.739522    5678 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69bf713b7aeaaa5ad79eecabdc6dc749006b4cfb86171ca0ce76103b3ffb5bf5\": container with ID starting with 69bf713b7aeaaa5ad79eecabdc6dc749006b4cfb86171ca0ce76103b3ffb5bf5 not found: ID does not exist" containerID="69bf713b7aeaaa5ad79eecabdc6dc749006b4cfb86171ca0ce76103b3ffb5bf5"
Oct 30 22:37:49.739619 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:37:49.739548    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69bf713b7aeaaa5ad79eecabdc6dc749006b4cfb86171ca0ce76103b3ffb5bf5"} err="failed to get container status \"69bf713b7aeaaa5ad79eecabdc6dc749006b4cfb86171ca0ce76103b3ffb5bf5\": rpc error: code = NotFound desc = could not find container \"69bf713b7aeaaa5ad79eecabdc6dc749006b4cfb86171ca0ce76103b3ffb5bf5\": container with ID starting with 69bf713b7aeaaa5ad79eecabdc6dc749006b4cfb86171ca0ce76103b3ffb5bf5 not found: ID does not exist"
Oct 30 22:37:49.817353 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:37:49.817314    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8157ccd-c3b7-4b4b-8dcb-9a8881eaff26-tls-certs\" (UniqueName: \"kubernetes.io/secret/c8157ccd-c3b7-4b4b-8dcb-9a8881eaff26-tls-certs\") pod \"c8157ccd-c3b7-4b4b-8dcb-9a8881eaff26\" (UID: \"c8157ccd-c3b7-4b4b-8dcb-9a8881eaff26\") "
Oct 30 22:37:49.817550 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:37:49.817363    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8157ccd-c3b7-4b4b-8dcb-9a8881eaff26-home\" (UniqueName: \"kubernetes.io/empty-dir/c8157ccd-c3b7-4b4b-8dcb-9a8881eaff26-home\") pod \"c8157ccd-c3b7-4b4b-8dcb-9a8881eaff26\" (UID: \"c8157ccd-c3b7-4b4b-8dcb-9a8881eaff26\") "
Oct 30 22:37:49.817550 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:37:49.817391    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8157ccd-c3b7-4b4b-8dcb-9a8881eaff26-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8157ccd-c3b7-4b4b-8dcb-9a8881eaff26-dshm\") pod \"c8157ccd-c3b7-4b4b-8dcb-9a8881eaff26\" (UID: \"c8157ccd-c3b7-4b4b-8dcb-9a8881eaff26\") "
Oct 30 22:37:49.817550 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:37:49.817446    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8157ccd-c3b7-4b4b-8dcb-9a8881eaff26-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8157ccd-c3b7-4b4b-8dcb-9a8881eaff26-tmp-dir\") pod \"c8157ccd-c3b7-4b4b-8dcb-9a8881eaff26\" (UID: \"c8157ccd-c3b7-4b4b-8dcb-9a8881eaff26\") "
Oct 30 22:37:49.817550 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:37:49.817491    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8157ccd-c3b7-4b4b-8dcb-9a8881eaff26-kube-api-access-5jj5l\" (UniqueName: \"kubernetes.io/projected/c8157ccd-c3b7-4b4b-8dcb-9a8881eaff26-kube-api-access-5jj5l\") pod \"c8157ccd-c3b7-4b4b-8dcb-9a8881eaff26\" (UID: \"c8157ccd-c3b7-4b4b-8dcb-9a8881eaff26\") "
Oct 30 22:37:49.817786 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:37:49.817561    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8157ccd-c3b7-4b4b-8dcb-9a8881eaff26-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8157ccd-c3b7-4b4b-8dcb-9a8881eaff26-model-cache\") pod \"c8157ccd-c3b7-4b4b-8dcb-9a8881eaff26\" (UID: \"c8157ccd-c3b7-4b4b-8dcb-9a8881eaff26\") "
Oct 30 22:37:49.817786 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:37:49.817650    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8157ccd-c3b7-4b4b-8dcb-9a8881eaff26-home" pod "c8157ccd-c3b7-4b4b-8dcb-9a8881eaff26" (UID: "c8157ccd-c3b7-4b4b-8dcb-9a8881eaff26"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:37:49.817907 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:37:49.817884    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8157ccd-c3b7-4b4b-8dcb-9a8881eaff26-model-cache" pod "c8157ccd-c3b7-4b4b-8dcb-9a8881eaff26" (UID: "c8157ccd-c3b7-4b4b-8dcb-9a8881eaff26"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:37:49.818029 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:37:49.818012    5678 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8157ccd-c3b7-4b4b-8dcb-9a8881eaff26-home\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:37:49.818096 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:37:49.818017    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8157ccd-c3b7-4b4b-8dcb-9a8881eaff26-tmp-dir" pod "c8157ccd-c3b7-4b4b-8dcb-9a8881eaff26" (UID: "c8157ccd-c3b7-4b4b-8dcb-9a8881eaff26"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:37:49.818096 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:37:49.818030    5678 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8157ccd-c3b7-4b4b-8dcb-9a8881eaff26-model-cache\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:37:49.819738 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:37:49.819703    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8157ccd-c3b7-4b4b-8dcb-9a8881eaff26-tls-certs" pod "c8157ccd-c3b7-4b4b-8dcb-9a8881eaff26" (UID: "c8157ccd-c3b7-4b4b-8dcb-9a8881eaff26"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:37:49.819874 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:37:49.819855    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8157ccd-c3b7-4b4b-8dcb-9a8881eaff26-kube-api-access-5jj5l" pod "c8157ccd-c3b7-4b4b-8dcb-9a8881eaff26" (UID: "c8157ccd-c3b7-4b4b-8dcb-9a8881eaff26"). InnerVolumeSpecName "kube-api-access-5jj5l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:37:49.819950 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:37:49.819932    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8157ccd-c3b7-4b4b-8dcb-9a8881eaff26-dshm" pod "c8157ccd-c3b7-4b4b-8dcb-9a8881eaff26" (UID: "c8157ccd-c3b7-4b4b-8dcb-9a8881eaff26"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:37:49.919257 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:37:49.919228    5678 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8157ccd-c3b7-4b4b-8dcb-9a8881eaff26-tls-certs\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:37:49.919257 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:37:49.919252    5678 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8157ccd-c3b7-4b4b-8dcb-9a8881eaff26-dshm\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:37:49.919257 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:37:49.919264    5678 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8157ccd-c3b7-4b4b-8dcb-9a8881eaff26-tmp-dir\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:37:49.919507 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:37:49.919273    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jj5l\" (UniqueName: \"kubernetes.io/projected/c8157ccd-c3b7-4b4b-8dcb-9a8881eaff26-kube-api-access-5jj5l\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:37:50.147488 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:37:50.147454    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-582593/test-model-582593-kserve-858995b55-6p46x"]
Oct 30 22:37:50.155396 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:37:50.155369    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-582593/test-model-582593-kserve-858995b55-6p46x"]
Oct 30 22:37:50.524421 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:37:50.524356    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8157ccd-c3b7-4b4b-8dcb-9a8881eaff26" path="/var/lib/kubelet/pods/c8157ccd-c3b7-4b4b-8dcb-9a8881eaff26/volumes"
Oct 30 22:38:10.572305 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:38:10.572263    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 22:38:14.132844 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:38:14.132812    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-582593-748c54b589-7kjl4"]
Oct 30 22:38:19.133715 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:38:19.133652    5678 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-582593-748c54b589-7kjl4" podUID="b1db85b2-f8d2-42b9-8731-4e9e9bff3f38" containerName="payload-processing" containerID="cri-o://4a802be0a5fa7992a0744ffa341d40593c2f85b641575dee1ff897a3e214cb25" gracePeriod=25
Oct 30 22:38:19.395218 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:38:19.395163    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-582593-748c54b589-7kjl4"
Oct 30 22:38:19.529069 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:38:19.529040    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1db85b2-f8d2-42b9-8731-4e9e9bff3f38-kube-api-access-rvxvb\" (UniqueName: \"kubernetes.io/projected/b1db85b2-f8d2-42b9-8731-4e9e9bff3f38-kube-api-access-rvxvb\") pod \"b1db85b2-f8d2-42b9-8731-4e9e9bff3f38\" (UID: \"b1db85b2-f8d2-42b9-8731-4e9e9bff3f38\") "
Oct 30 22:38:19.529229 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:38:19.529150    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1db85b2-f8d2-42b9-8731-4e9e9bff3f38-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1db85b2-f8d2-42b9-8731-4e9e9bff3f38-plugins-config-volume\") pod \"b1db85b2-f8d2-42b9-8731-4e9e9bff3f38\" (UID: \"b1db85b2-f8d2-42b9-8731-4e9e9bff3f38\") "
Oct 30 22:38:19.529526 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:38:19.529502    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1db85b2-f8d2-42b9-8731-4e9e9bff3f38-plugins-config-volume" pod "b1db85b2-f8d2-42b9-8731-4e9e9bff3f38" (UID: "b1db85b2-f8d2-42b9-8731-4e9e9bff3f38"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:38:19.529690 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:38:19.529675    5678 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1db85b2-f8d2-42b9-8731-4e9e9bff3f38-plugins-config-volume\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:38:19.531132 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:38:19.531114    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1db85b2-f8d2-42b9-8731-4e9e9bff3f38-kube-api-access-rvxvb" pod "b1db85b2-f8d2-42b9-8731-4e9e9bff3f38" (UID: "b1db85b2-f8d2-42b9-8731-4e9e9bff3f38"). InnerVolumeSpecName "kube-api-access-rvxvb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:38:19.630940 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:38:19.630902    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvxvb\" (UniqueName: \"kubernetes.io/projected/b1db85b2-f8d2-42b9-8731-4e9e9bff3f38-kube-api-access-rvxvb\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:38:19.903879 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:38:19.903842    5678 generic.go:356] "Generic (PLEG): container finished" podID="b1db85b2-f8d2-42b9-8731-4e9e9bff3f38" containerID="4a802be0a5fa7992a0744ffa341d40593c2f85b641575dee1ff897a3e214cb25" exitCode=0
Oct 30 22:38:19.904060 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:38:19.903888    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-582593-748c54b589-7kjl4"
Oct 30 22:38:19.904060 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:38:19.903928    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-582593-748c54b589-7kjl4" event={"ID":"b1db85b2-f8d2-42b9-8731-4e9e9bff3f38","Type":"ContainerDied","Data":"4a802be0a5fa7992a0744ffa341d40593c2f85b641575dee1ff897a3e214cb25"}
Oct 30 22:38:19.904060 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:38:19.903988    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-582593-748c54b589-7kjl4" event={"ID":"b1db85b2-f8d2-42b9-8731-4e9e9bff3f38","Type":"ContainerDied","Data":"f004f14d5155b55de750313d6206d9199b8a0b842231db206ff5fc521c62b9e5"}
Oct 30 22:38:19.904060 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:38:19.904020    5678 scope.go:122] "RemoveContainer" containerID="4a802be0a5fa7992a0744ffa341d40593c2f85b641575dee1ff897a3e214cb25"
Oct 30 22:38:19.913963 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:38:19.913943    5678 scope.go:122] "RemoveContainer" containerID="4a802be0a5fa7992a0744ffa341d40593c2f85b641575dee1ff897a3e214cb25"
Oct 30 22:38:19.914240 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:38:19.914222    5678 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a802be0a5fa7992a0744ffa341d40593c2f85b641575dee1ff897a3e214cb25\": container with ID starting with 4a802be0a5fa7992a0744ffa341d40593c2f85b641575dee1ff897a3e214cb25 not found: ID does not exist" containerID="4a802be0a5fa7992a0744ffa341d40593c2f85b641575dee1ff897a3e214cb25"
Oct 30 22:38:19.914283 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:38:19.914247    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a802be0a5fa7992a0744ffa341d40593c2f85b641575dee1ff897a3e214cb25"} err="failed to get container status \"4a802be0a5fa7992a0744ffa341d40593c2f85b641575dee1ff897a3e214cb25\": rpc error: code = NotFound desc = could not find container \"4a802be0a5fa7992a0744ffa341d40593c2f85b641575dee1ff897a3e214cb25\": container with ID starting with 4a802be0a5fa7992a0744ffa341d40593c2f85b641575dee1ff897a3e214cb25 not found: ID does not exist"
Oct 30 22:38:20.063180 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:38:20.063148    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-582593-748c54b589-7kjl4"]
Oct 30 22:38:20.070436 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:38:20.070394    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-582593-748c54b589-7kjl4"]
Oct 30 22:38:20.523101 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:38:20.523064    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1db85b2-f8d2-42b9-8731-4e9e9bff3f38" path="/var/lib/kubelet/pods/b1db85b2-f8d2-42b9-8731-4e9e9bff3f38/volumes"
Oct 30 22:38:33.547806 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:38:33.547731    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-582593-74c6898768-vr64l"]
Oct 30 22:38:33.550279 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:38:33.547938    5678 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-582593-74c6898768-vr64l" podUID="09ddef89-ac05-4e76-b7ef-449236df8900" containerName="maas-api" containerID="cri-o://73e1dd2961c02d67fbf1ef227d22d1e5a9c121e012b84b4d7d34bc18655f9fc4" gracePeriod=30
Oct 30 22:38:33.805006 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:38:33.804947    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-582593-74c6898768-vr64l"
Oct 30 22:38:33.880155 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:38:33.880121    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09ddef89-ac05-4e76-b7ef-449236df8900-kube-api-access-gw7n8\" (UniqueName: \"kubernetes.io/projected/09ddef89-ac05-4e76-b7ef-449236df8900-kube-api-access-gw7n8\") pod \"09ddef89-ac05-4e76-b7ef-449236df8900\" (UID: \"09ddef89-ac05-4e76-b7ef-449236df8900\") "
Oct 30 22:38:33.880346 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:38:33.880210    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09ddef89-ac05-4e76-b7ef-449236df8900-metrics-certs\" (UniqueName: \"kubernetes.io/secret/09ddef89-ac05-4e76-b7ef-449236df8900-metrics-certs\") pod \"09ddef89-ac05-4e76-b7ef-449236df8900\" (UID: \"09ddef89-ac05-4e76-b7ef-449236df8900\") "
Oct 30 22:38:33.880346 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:38:33.880263    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09ddef89-ac05-4e76-b7ef-449236df8900-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/09ddef89-ac05-4e76-b7ef-449236df8900-maas-api-tls\") pod \"09ddef89-ac05-4e76-b7ef-449236df8900\" (UID: \"09ddef89-ac05-4e76-b7ef-449236df8900\") "
Oct 30 22:38:33.880346 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:38:33.880306    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09ddef89-ac05-4e76-b7ef-449236df8900-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/09ddef89-ac05-4e76-b7ef-449236df8900-openshift-service-ca\") pod \"09ddef89-ac05-4e76-b7ef-449236df8900\" (UID: \"09ddef89-ac05-4e76-b7ef-449236df8900\") "
Oct 30 22:38:33.880765 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:38:33.880734    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09ddef89-ac05-4e76-b7ef-449236df8900-openshift-service-ca" pod "09ddef89-ac05-4e76-b7ef-449236df8900" (UID: "09ddef89-ac05-4e76-b7ef-449236df8900"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:38:33.882359 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:38:33.882329    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09ddef89-ac05-4e76-b7ef-449236df8900-metrics-certs" pod "09ddef89-ac05-4e76-b7ef-449236df8900" (UID: "09ddef89-ac05-4e76-b7ef-449236df8900"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:38:33.882359 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:38:33.882335    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09ddef89-ac05-4e76-b7ef-449236df8900-kube-api-access-gw7n8" pod "09ddef89-ac05-4e76-b7ef-449236df8900" (UID: "09ddef89-ac05-4e76-b7ef-449236df8900"). InnerVolumeSpecName "kube-api-access-gw7n8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:38:33.882532 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:38:33.882365    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09ddef89-ac05-4e76-b7ef-449236df8900-maas-api-tls" pod "09ddef89-ac05-4e76-b7ef-449236df8900" (UID: "09ddef89-ac05-4e76-b7ef-449236df8900"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:38:33.982149 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:38:33.982112    5678 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/09ddef89-ac05-4e76-b7ef-449236df8900-metrics-certs\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:38:33.982149 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:38:33.982145    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/09ddef89-ac05-4e76-b7ef-449236df8900-maas-api-tls\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:38:33.982349 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:38:33.982159    5678 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/09ddef89-ac05-4e76-b7ef-449236df8900-openshift-service-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:38:33.982349 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:38:33.982172    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gw7n8\" (UniqueName: \"kubernetes.io/projected/09ddef89-ac05-4e76-b7ef-449236df8900-kube-api-access-gw7n8\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:38:34.003470 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:38:34.003434    5678 generic.go:356] "Generic (PLEG): container finished" podID="09ddef89-ac05-4e76-b7ef-449236df8900" containerID="73e1dd2961c02d67fbf1ef227d22d1e5a9c121e012b84b4d7d34bc18655f9fc4" exitCode=0
Oct 30 22:38:34.003603 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:38:34.003475    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-582593-74c6898768-vr64l"
Oct 30 22:38:34.003603 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:38:34.003503    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-582593-74c6898768-vr64l" event={"ID":"09ddef89-ac05-4e76-b7ef-449236df8900","Type":"ContainerDied","Data":"73e1dd2961c02d67fbf1ef227d22d1e5a9c121e012b84b4d7d34bc18655f9fc4"}
Oct 30 22:38:34.003603 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:38:34.003553    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-582593-74c6898768-vr64l" event={"ID":"09ddef89-ac05-4e76-b7ef-449236df8900","Type":"ContainerDied","Data":"1d140501fcdd13b9ae73cd3b236a5353b8ecbab6d18d3c9218d1d519698c15f7"}
Oct 30 22:38:34.003603 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:38:34.003582    5678 scope.go:122] "RemoveContainer" containerID="73e1dd2961c02d67fbf1ef227d22d1e5a9c121e012b84b4d7d34bc18655f9fc4"
Oct 30 22:38:34.014391 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:38:34.014372    5678 scope.go:122] "RemoveContainer" containerID="73e1dd2961c02d67fbf1ef227d22d1e5a9c121e012b84b4d7d34bc18655f9fc4"
Oct 30 22:38:34.014674 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:38:34.014655    5678 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73e1dd2961c02d67fbf1ef227d22d1e5a9c121e012b84b4d7d34bc18655f9fc4\": container with ID starting with 73e1dd2961c02d67fbf1ef227d22d1e5a9c121e012b84b4d7d34bc18655f9fc4 not found: ID does not exist" containerID="73e1dd2961c02d67fbf1ef227d22d1e5a9c121e012b84b4d7d34bc18655f9fc4"
Oct 30 22:38:34.014739 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:38:34.014685    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73e1dd2961c02d67fbf1ef227d22d1e5a9c121e012b84b4d7d34bc18655f9fc4"} err="failed to get container status \"73e1dd2961c02d67fbf1ef227d22d1e5a9c121e012b84b4d7d34bc18655f9fc4\": rpc error: code = NotFound desc = could not find container \"73e1dd2961c02d67fbf1ef227d22d1e5a9c121e012b84b4d7d34bc18655f9fc4\": container with ID starting with 73e1dd2961c02d67fbf1ef227d22d1e5a9c121e012b84b4d7d34bc18655f9fc4 not found: ID does not exist"
Oct 30 22:38:34.160808 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:38:34.160778    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-582593-74c6898768-vr64l"]
Oct 30 22:38:34.167198 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:38:34.167174    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-582593-74c6898768-vr64l"]
Oct 30 22:38:34.523758 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:38:34.523685    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09ddef89-ac05-4e76-b7ef-449236df8900" path="/var/lib/kubelet/pods/09ddef89-ac05-4e76-b7ef-449236df8900/volumes"
Oct 30 22:38:40.578325 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:38:40.578284    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 22:38:43.196354 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:38:43.196323    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-582593-openshift-default-868f8d8b7-w97qr"]
Oct 30 22:38:43.196759 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:38:43.196528    5678 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-582593-openshift-default-868f8d8b7-w97qr" podUID="e325caa7-d4fb-4a0e-9fe0-ee95742278b3" containerName="istio-proxy" containerID="cri-o://05571cd067d9e4789400a35ab789934ec11012ee436b0285ac8626731519a9eb" gracePeriod=30
Oct 30 22:38:48.450655 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:38:48.450633    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-582593-openshift-default-868f8d8b7-w97qr"
Oct 30 22:38:48.531350 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:38:48.531282    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e325caa7-d4fb-4a0e-9fe0-ee95742278b3-kube-api-access-rsfrx\" (UniqueName: \"kubernetes.io/projected/e325caa7-d4fb-4a0e-9fe0-ee95742278b3-kube-api-access-rsfrx\") pod \"e325caa7-d4fb-4a0e-9fe0-ee95742278b3\" (UID: \"e325caa7-d4fb-4a0e-9fe0-ee95742278b3\") "
Oct 30 22:38:48.531350 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:38:48.531331    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e325caa7-d4fb-4a0e-9fe0-ee95742278b3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e325caa7-d4fb-4a0e-9fe0-ee95742278b3-istio-envoy\") pod \"e325caa7-d4fb-4a0e-9fe0-ee95742278b3\" (UID: \"e325caa7-d4fb-4a0e-9fe0-ee95742278b3\") "
Oct 30 22:38:48.531570 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:38:48.531352    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e325caa7-d4fb-4a0e-9fe0-ee95742278b3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e325caa7-d4fb-4a0e-9fe0-ee95742278b3-istiod-ca-cert\") pod \"e325caa7-d4fb-4a0e-9fe0-ee95742278b3\" (UID: \"e325caa7-d4fb-4a0e-9fe0-ee95742278b3\") "
Oct 30 22:38:48.531570 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:38:48.531370    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e325caa7-d4fb-4a0e-9fe0-ee95742278b3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e325caa7-d4fb-4a0e-9fe0-ee95742278b3-workload-certs\") pod \"e325caa7-d4fb-4a0e-9fe0-ee95742278b3\" (UID: \"e325caa7-d4fb-4a0e-9fe0-ee95742278b3\") "
Oct 30 22:38:48.531570 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:38:48.531484    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e325caa7-d4fb-4a0e-9fe0-ee95742278b3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e325caa7-d4fb-4a0e-9fe0-ee95742278b3-workload-socket\") pod \"e325caa7-d4fb-4a0e-9fe0-ee95742278b3\" (UID: \"e325caa7-d4fb-4a0e-9fe0-ee95742278b3\") "
Oct 30 22:38:48.531570 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:38:48.531528    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e325caa7-d4fb-4a0e-9fe0-ee95742278b3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e325caa7-d4fb-4a0e-9fe0-ee95742278b3-credential-socket\") pod \"e325caa7-d4fb-4a0e-9fe0-ee95742278b3\" (UID: \"e325caa7-d4fb-4a0e-9fe0-ee95742278b3\") "
Oct 30 22:38:48.531570 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:38:48.531559    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e325caa7-d4fb-4a0e-9fe0-ee95742278b3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e325caa7-d4fb-4a0e-9fe0-ee95742278b3-istio-data\") pod \"e325caa7-d4fb-4a0e-9fe0-ee95742278b3\" (UID: \"e325caa7-d4fb-4a0e-9fe0-ee95742278b3\") "
Oct 30 22:38:48.531851 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:38:48.531637    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e325caa7-d4fb-4a0e-9fe0-ee95742278b3-istio-token\" (UniqueName: \"kubernetes.io/projected/e325caa7-d4fb-4a0e-9fe0-ee95742278b3-istio-token\") pod \"e325caa7-d4fb-4a0e-9fe0-ee95742278b3\" (UID: \"e325caa7-d4fb-4a0e-9fe0-ee95742278b3\") "
Oct 30 22:38:48.531851 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:38:48.531641    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e325caa7-d4fb-4a0e-9fe0-ee95742278b3-workload-certs" pod "e325caa7-d4fb-4a0e-9fe0-ee95742278b3" (UID: "e325caa7-d4fb-4a0e-9fe0-ee95742278b3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:38:48.531851 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:38:48.531789    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e325caa7-d4fb-4a0e-9fe0-ee95742278b3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e325caa7-d4fb-4a0e-9fe0-ee95742278b3-istio-podinfo\") pod \"e325caa7-d4fb-4a0e-9fe0-ee95742278b3\" (UID: \"e325caa7-d4fb-4a0e-9fe0-ee95742278b3\") "
Oct 30 22:38:48.531851 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:38:48.531797    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e325caa7-d4fb-4a0e-9fe0-ee95742278b3-credential-socket" pod "e325caa7-d4fb-4a0e-9fe0-ee95742278b3" (UID: "e325caa7-d4fb-4a0e-9fe0-ee95742278b3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:38:48.532071 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:38:48.531816    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e325caa7-d4fb-4a0e-9fe0-ee95742278b3-istiod-ca-cert" pod "e325caa7-d4fb-4a0e-9fe0-ee95742278b3" (UID: "e325caa7-d4fb-4a0e-9fe0-ee95742278b3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:38:48.532071 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:38:48.531910    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e325caa7-d4fb-4a0e-9fe0-ee95742278b3-istio-data" pod "e325caa7-d4fb-4a0e-9fe0-ee95742278b3" (UID: "e325caa7-d4fb-4a0e-9fe0-ee95742278b3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:38:48.532071 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:38:48.531893    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e325caa7-d4fb-4a0e-9fe0-ee95742278b3-workload-socket" pod "e325caa7-d4fb-4a0e-9fe0-ee95742278b3" (UID: "e325caa7-d4fb-4a0e-9fe0-ee95742278b3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:38:48.532793 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:38:48.532346    5678 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e325caa7-d4fb-4a0e-9fe0-ee95742278b3-istiod-ca-cert\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:38:48.532793 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:38:48.532689    5678 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e325caa7-d4fb-4a0e-9fe0-ee95742278b3-workload-certs\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:38:48.532793 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:38:48.532711    5678 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e325caa7-d4fb-4a0e-9fe0-ee95742278b3-workload-socket\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:38:48.532793 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:38:48.532729    5678 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e325caa7-d4fb-4a0e-9fe0-ee95742278b3-credential-socket\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:38:48.532793 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:38:48.532744    5678 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e325caa7-d4fb-4a0e-9fe0-ee95742278b3-istio-data\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:38:48.533826 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:38:48.533801    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e325caa7-d4fb-4a0e-9fe0-ee95742278b3-istio-token" pod "e325caa7-d4fb-4a0e-9fe0-ee95742278b3" (UID: "e325caa7-d4fb-4a0e-9fe0-ee95742278b3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:38:48.533928 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:38:48.533842    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e325caa7-d4fb-4a0e-9fe0-ee95742278b3-istio-envoy" pod "e325caa7-d4fb-4a0e-9fe0-ee95742278b3" (UID: "e325caa7-d4fb-4a0e-9fe0-ee95742278b3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:38:48.533928 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:38:48.533870    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e325caa7-d4fb-4a0e-9fe0-ee95742278b3-kube-api-access-rsfrx" pod "e325caa7-d4fb-4a0e-9fe0-ee95742278b3" (UID: "e325caa7-d4fb-4a0e-9fe0-ee95742278b3"). InnerVolumeSpecName "kube-api-access-rsfrx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:38:48.534005 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:38:48.533926    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e325caa7-d4fb-4a0e-9fe0-ee95742278b3-istio-podinfo" pod "e325caa7-d4fb-4a0e-9fe0-ee95742278b3" (UID: "e325caa7-d4fb-4a0e-9fe0-ee95742278b3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:38:48.634198 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:38:48.634162    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rsfrx\" (UniqueName: \"kubernetes.io/projected/e325caa7-d4fb-4a0e-9fe0-ee95742278b3-kube-api-access-rsfrx\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:38:48.634198 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:38:48.634190    5678 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e325caa7-d4fb-4a0e-9fe0-ee95742278b3-istio-envoy\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:38:48.634379 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:38:48.634204    5678 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e325caa7-d4fb-4a0e-9fe0-ee95742278b3-istio-token\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:38:48.634379 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:38:48.634218    5678 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e325caa7-d4fb-4a0e-9fe0-ee95742278b3-istio-podinfo\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:38:49.090608 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:38:49.090573    5678 generic.go:356] "Generic (PLEG): container finished" podID="e325caa7-d4fb-4a0e-9fe0-ee95742278b3" containerID="05571cd067d9e4789400a35ab789934ec11012ee436b0285ac8626731519a9eb" exitCode=0
Oct 30 22:38:49.090786 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:38:49.090632    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-582593-openshift-default-868f8d8b7-w97qr"
Oct 30 22:38:49.090786 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:38:49.090659    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-582593-openshift-default-868f8d8b7-w97qr" event={"ID":"e325caa7-d4fb-4a0e-9fe0-ee95742278b3","Type":"ContainerDied","Data":"05571cd067d9e4789400a35ab789934ec11012ee436b0285ac8626731519a9eb"}
Oct 30 22:38:49.090786 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:38:49.090720    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-582593-openshift-default-868f8d8b7-w97qr" event={"ID":"e325caa7-d4fb-4a0e-9fe0-ee95742278b3","Type":"ContainerDied","Data":"a1e254a1f816b22abb5964e9fdf8c0e1a38f2b6f3cd6ab9295d1adbfac957881"}
Oct 30 22:38:49.090786 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:38:49.090750    5678 scope.go:122] "RemoveContainer" containerID="05571cd067d9e4789400a35ab789934ec11012ee436b0285ac8626731519a9eb"
Oct 30 22:38:49.100849 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:38:49.100827    5678 scope.go:122] "RemoveContainer" containerID="bd50cf584c9705bd333a3688197936c0b11b4611c0c209d3b004442718f52b1e"
Oct 30 22:38:49.109339 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:38:49.109322    5678 scope.go:122] "RemoveContainer" containerID="05571cd067d9e4789400a35ab789934ec11012ee436b0285ac8626731519a9eb"
Oct 30 22:38:49.109604 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:38:49.109586    5678 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05571cd067d9e4789400a35ab789934ec11012ee436b0285ac8626731519a9eb\": container with ID starting with 05571cd067d9e4789400a35ab789934ec11012ee436b0285ac8626731519a9eb not found: ID does not exist" containerID="05571cd067d9e4789400a35ab789934ec11012ee436b0285ac8626731519a9eb"
Oct 30 22:38:49.109649 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:38:49.109612    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05571cd067d9e4789400a35ab789934ec11012ee436b0285ac8626731519a9eb"} err="failed to get container status \"05571cd067d9e4789400a35ab789934ec11012ee436b0285ac8626731519a9eb\": rpc error: code = NotFound desc = could not find container \"05571cd067d9e4789400a35ab789934ec11012ee436b0285ac8626731519a9eb\": container with ID starting with 05571cd067d9e4789400a35ab789934ec11012ee436b0285ac8626731519a9eb not found: ID does not exist"
Oct 30 22:38:49.109649 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:38:49.109629    5678 scope.go:122] "RemoveContainer" containerID="bd50cf584c9705bd333a3688197936c0b11b4611c0c209d3b004442718f52b1e"
Oct 30 22:38:49.109849 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:38:49.109830    5678 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd50cf584c9705bd333a3688197936c0b11b4611c0c209d3b004442718f52b1e\": container with ID starting with bd50cf584c9705bd333a3688197936c0b11b4611c0c209d3b004442718f52b1e not found: ID does not exist" containerID="bd50cf584c9705bd333a3688197936c0b11b4611c0c209d3b004442718f52b1e"
Oct 30 22:38:49.109896 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:38:49.109854    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd50cf584c9705bd333a3688197936c0b11b4611c0c209d3b004442718f52b1e"} err="failed to get container status \"bd50cf584c9705bd333a3688197936c0b11b4611c0c209d3b004442718f52b1e\": rpc error: code = NotFound desc = could not find container \"bd50cf584c9705bd333a3688197936c0b11b4611c0c209d3b004442718f52b1e\": container with ID starting with bd50cf584c9705bd333a3688197936c0b11b4611c0c209d3b004442718f52b1e not found: ID does not exist"
Oct 30 22:38:49.248643 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:38:49.248618    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-582593-openshift-default-868f8d8b7-w97qr"]
Oct 30 22:38:49.295464 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:38:49.295433    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-582593-openshift-default-868f8d8b7-w97qr"]
Oct 30 22:38:50.522457 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:38:50.522420    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e325caa7-d4fb-4a0e-9fe0-ee95742278b3" path="/var/lib/kubelet/pods/e325caa7-d4fb-4a0e-9fe0-ee95742278b3/volumes"
Oct 30 22:38:54.571997 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:38:54.571956    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-506964-7dc79f55ff-wm696"]
Oct 30 22:39:00.127122 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:00.127073    5678 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-506964-7dc79f55ff-wm696" podUID="f1e95a6d-29c6-4c95-b537-1037089dcab8" containerName="payload-pre-processing" containerID="cri-o://11e5d17edc65c348579a6d9a79634fdd742105d6cc8c7f8c7c3bcc9921397e67" gracePeriod=25
Oct 30 22:39:08.353579 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:08.353538    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-506964-7dc79f55ff-wm696" podUID="f1e95a6d-29c6-4c95-b537-1037089dcab8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:39:10.570961 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:39:10.570920    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 22:39:17.329993 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:17.329951    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-75fb62-5649854489-92pfg"]
Oct 30 22:39:17.429630 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:17.429598    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-75fb62-57ff7485f9-fl2h5"]
Oct 30 22:39:18.353675 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:18.353641    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-506964-7dc79f55ff-wm696" podUID="f1e95a6d-29c6-4c95-b537-1037089dcab8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:39:20.281184 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:20.281143    5678 generic.go:356] "Generic (PLEG): container finished" podID="f1e95a6d-29c6-4c95-b537-1037089dcab8" containerID="11e5d17edc65c348579a6d9a79634fdd742105d6cc8c7f8c7c3bcc9921397e67" exitCode=0
Oct 30 22:39:20.281568 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:20.281216    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-506964-7dc79f55ff-wm696" event={"ID":"f1e95a6d-29c6-4c95-b537-1037089dcab8","Type":"ContainerDied","Data":"11e5d17edc65c348579a6d9a79634fdd742105d6cc8c7f8c7c3bcc9921397e67"}
Oct 30 22:39:20.389394 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:20.389340    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-506964-7dc79f55ff-wm696"
Oct 30 22:39:20.470695 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:20.470662    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1e95a6d-29c6-4c95-b537-1037089dcab8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1e95a6d-29c6-4c95-b537-1037089dcab8-plugins-config-volume\") pod \"f1e95a6d-29c6-4c95-b537-1037089dcab8\" (UID: \"f1e95a6d-29c6-4c95-b537-1037089dcab8\") "
Oct 30 22:39:20.470864 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:20.470757    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1e95a6d-29c6-4c95-b537-1037089dcab8-kube-api-access-8p2tw\" (UniqueName: \"kubernetes.io/projected/f1e95a6d-29c6-4c95-b537-1037089dcab8-kube-api-access-8p2tw\") pod \"f1e95a6d-29c6-4c95-b537-1037089dcab8\" (UID: \"f1e95a6d-29c6-4c95-b537-1037089dcab8\") "
Oct 30 22:39:20.471020 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:20.470995    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1e95a6d-29c6-4c95-b537-1037089dcab8-plugins-config-volume" pod "f1e95a6d-29c6-4c95-b537-1037089dcab8" (UID: "f1e95a6d-29c6-4c95-b537-1037089dcab8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:39:20.471129 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:20.471115    5678 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1e95a6d-29c6-4c95-b537-1037089dcab8-plugins-config-volume\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:39:20.472832 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:20.472814    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1e95a6d-29c6-4c95-b537-1037089dcab8-kube-api-access-8p2tw" pod "f1e95a6d-29c6-4c95-b537-1037089dcab8" (UID: "f1e95a6d-29c6-4c95-b537-1037089dcab8"). InnerVolumeSpecName "kube-api-access-8p2tw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:39:20.572702 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:20.572671    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8p2tw\" (UniqueName: \"kubernetes.io/projected/f1e95a6d-29c6-4c95-b537-1037089dcab8-kube-api-access-8p2tw\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:39:21.287755 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:21.287722    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-506964-7dc79f55ff-wm696"
Oct 30 22:39:21.288345 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:21.287747    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-506964-7dc79f55ff-wm696" event={"ID":"f1e95a6d-29c6-4c95-b537-1037089dcab8","Type":"ContainerDied","Data":"98df0c01beaa13055720b5b6a2561d9699bc5bbfdb20c92827e82694b31c55ff"}
Oct 30 22:39:21.288345 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:21.287831    5678 scope.go:122] "RemoveContainer" containerID="11e5d17edc65c348579a6d9a79634fdd742105d6cc8c7f8c7c3bcc9921397e67"
Oct 30 22:39:21.445363 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:21.445325    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-506964-7dc79f55ff-wm696"]
Oct 30 22:39:21.451228 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:21.451201    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-506964-7dc79f55ff-wm696"]
Oct 30 22:39:22.330918 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:22.330886    5678 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-75fb62-5649854489-92pfg" podUID="95ef4304-6677-4325-8396-31575317404c" containerName="payload-processing" containerID="cri-o://e04707050b9f01b8187bff71c74d10dcfd51ffa90ff37773891da0a9e02b6c7e" gracePeriod=25
Oct 30 22:39:22.430300 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:22.430248    5678 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-75fb62-57ff7485f9-fl2h5" podUID="01b7d4f9-77ed-4292-a572-3a910bc8521e" containerName="payload-pre-processing" containerID="cri-o://708b64baddc2018a602c41d286ea7bb816b507bc0119e5f035dd3bafa3d7336a" gracePeriod=25
Oct 30 22:39:22.434940 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:22.434913    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-506964-5d557d7b77-77m8h"]
Oct 30 22:39:22.435173 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:22.435149    5678 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-506964-5d557d7b77-77m8h" podUID="699c4653-4da2-4b6f-965e-d34b632a8340" containerName="maas-api" containerID="cri-o://5b70af7c2321d75355369d6883cc2824aa369ca459533f36e200cf462610883a" gracePeriod=30
Oct 30 22:39:22.523530 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:22.523500    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1e95a6d-29c6-4c95-b537-1037089dcab8" path="/var/lib/kubelet/pods/f1e95a6d-29c6-4c95-b537-1037089dcab8/volumes"
Oct 30 22:39:22.696853 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:22.696829    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-506964-5d557d7b77-77m8h"
Oct 30 22:39:22.793315 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:22.793284    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/699c4653-4da2-4b6f-965e-d34b632a8340-kube-api-access-vfrgk\" (UniqueName: \"kubernetes.io/projected/699c4653-4da2-4b6f-965e-d34b632a8340-kube-api-access-vfrgk\") pod \"699c4653-4da2-4b6f-965e-d34b632a8340\" (UID: \"699c4653-4da2-4b6f-965e-d34b632a8340\") "
Oct 30 22:39:22.793546 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:22.793348    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/699c4653-4da2-4b6f-965e-d34b632a8340-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/699c4653-4da2-4b6f-965e-d34b632a8340-openshift-service-ca\") pod \"699c4653-4da2-4b6f-965e-d34b632a8340\" (UID: \"699c4653-4da2-4b6f-965e-d34b632a8340\") "
Oct 30 22:39:22.793546 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:22.793395    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/699c4653-4da2-4b6f-965e-d34b632a8340-metrics-certs\" (UniqueName: \"kubernetes.io/secret/699c4653-4da2-4b6f-965e-d34b632a8340-metrics-certs\") pod \"699c4653-4da2-4b6f-965e-d34b632a8340\" (UID: \"699c4653-4da2-4b6f-965e-d34b632a8340\") "
Oct 30 22:39:22.793689 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:22.793560    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/699c4653-4da2-4b6f-965e-d34b632a8340-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/699c4653-4da2-4b6f-965e-d34b632a8340-maas-api-tls\") pod \"699c4653-4da2-4b6f-965e-d34b632a8340\" (UID: \"699c4653-4da2-4b6f-965e-d34b632a8340\") "
Oct 30 22:39:22.794279 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:22.794248    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/699c4653-4da2-4b6f-965e-d34b632a8340-openshift-service-ca" pod "699c4653-4da2-4b6f-965e-d34b632a8340" (UID: "699c4653-4da2-4b6f-965e-d34b632a8340"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:39:22.795769 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:22.795742    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/699c4653-4da2-4b6f-965e-d34b632a8340-metrics-certs" pod "699c4653-4da2-4b6f-965e-d34b632a8340" (UID: "699c4653-4da2-4b6f-965e-d34b632a8340"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:39:22.796068 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:22.796039    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/699c4653-4da2-4b6f-965e-d34b632a8340-maas-api-tls" pod "699c4653-4da2-4b6f-965e-d34b632a8340" (UID: "699c4653-4da2-4b6f-965e-d34b632a8340"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:39:22.801776 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:22.801751    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/699c4653-4da2-4b6f-965e-d34b632a8340-kube-api-access-vfrgk" pod "699c4653-4da2-4b6f-965e-d34b632a8340" (UID: "699c4653-4da2-4b6f-965e-d34b632a8340"). InnerVolumeSpecName "kube-api-access-vfrgk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:39:22.894812 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:22.894782    5678 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/699c4653-4da2-4b6f-965e-d34b632a8340-openshift-service-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:39:22.894812 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:22.894808    5678 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/699c4653-4da2-4b6f-965e-d34b632a8340-metrics-certs\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:39:22.895014 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:22.894819    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/699c4653-4da2-4b6f-965e-d34b632a8340-maas-api-tls\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:39:22.895014 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:22.894829    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfrgk\" (UniqueName: \"kubernetes.io/projected/699c4653-4da2-4b6f-965e-d34b632a8340-kube-api-access-vfrgk\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:39:23.314498 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:23.306386    5678 generic.go:356] "Generic (PLEG): container finished" podID="699c4653-4da2-4b6f-965e-d34b632a8340" containerID="5b70af7c2321d75355369d6883cc2824aa369ca459533f36e200cf462610883a" exitCode=0
Oct 30 22:39:23.314498 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:23.306621    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-506964-5d557d7b77-77m8h" event={"ID":"699c4653-4da2-4b6f-965e-d34b632a8340","Type":"ContainerDied","Data":"5b70af7c2321d75355369d6883cc2824aa369ca459533f36e200cf462610883a"}
Oct 30 22:39:23.314498 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:23.306673    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-506964-5d557d7b77-77m8h" event={"ID":"699c4653-4da2-4b6f-965e-d34b632a8340","Type":"ContainerDied","Data":"ee6be2ce0f37db8201f8edbbf96c238cd3b9bf479083d6d3534d50d9e877ce38"}
Oct 30 22:39:23.314498 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:23.306704    5678 scope.go:122] "RemoveContainer" containerID="5b70af7c2321d75355369d6883cc2824aa369ca459533f36e200cf462610883a"
Oct 30 22:39:23.314498 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:23.306819    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-506964-5d557d7b77-77m8h"
Oct 30 22:39:23.330680 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:23.330643    5678 scope.go:122] "RemoveContainer" containerID="5b70af7c2321d75355369d6883cc2824aa369ca459533f36e200cf462610883a"
Oct 30 22:39:23.331021 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:39:23.330993    5678 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b70af7c2321d75355369d6883cc2824aa369ca459533f36e200cf462610883a\": container with ID starting with 5b70af7c2321d75355369d6883cc2824aa369ca459533f36e200cf462610883a not found: ID does not exist" containerID="5b70af7c2321d75355369d6883cc2824aa369ca459533f36e200cf462610883a"
Oct 30 22:39:23.331296 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:23.331032    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b70af7c2321d75355369d6883cc2824aa369ca459533f36e200cf462610883a"} err="failed to get container status \"5b70af7c2321d75355369d6883cc2824aa369ca459533f36e200cf462610883a\": rpc error: code = NotFound desc = could not find container \"5b70af7c2321d75355369d6883cc2824aa369ca459533f36e200cf462610883a\": container with ID starting with 5b70af7c2321d75355369d6883cc2824aa369ca459533f36e200cf462610883a not found: ID does not exist"
Oct 30 22:39:23.473256 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:23.473218    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-506964-5d557d7b77-77m8h"]
Oct 30 22:39:23.477895 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:23.477868    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-506964-5d557d7b77-77m8h"]
Oct 30 22:39:24.522392 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:24.522316    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="699c4653-4da2-4b6f-965e-d34b632a8340" path="/var/lib/kubelet/pods/699c4653-4da2-4b6f-965e-d34b632a8340/volumes"
Oct 30 22:39:25.830053 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:25.830012    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-75fb62-57ff7485f9-fl2h5" podUID="01b7d4f9-77ed-4292-a572-3a910bc8521e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:39:25.830508 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:25.830023    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-75fb62-5649854489-92pfg" podUID="95ef4304-6677-4325-8396-31575317404c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:39:31.590261 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:31.590227    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-506964-openshift-default-5bfdbc5849-4qwwq"]
Oct 30 22:39:31.590814 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:31.590470    5678 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-506964-openshift-default-5bfdbc5849-4qwwq" podUID="64a0ba20-2130-4df1-b147-28a52a7d0e62" containerName="istio-proxy" containerID="cri-o://2a2a0c368ccd60107ea46ad21cd211e2bdf53734db49f376966c0db909f47426" gracePeriod=30
Oct 30 22:39:31.759420 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:31.759370    5678 patch_prober.go:29] interesting pod/e2e-inf-b-506964-openshift-default-5bfdbc5849-4qwwq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:39:31.759581 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:31.759442    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-506964-openshift-default-5bfdbc5849-4qwwq" podUID="64a0ba20-2130-4df1-b147-28a52a7d0e62" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:39:33.181842 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:33.181804    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-75fb62-845c55c77b-gmd6f"]
Oct 30 22:39:33.182259 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:33.182001    5678 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-75fb62-845c55c77b-gmd6f" podUID="0e40fbc5-668a-4917-9344-dea69c625d26" containerName="maas-api" containerID="cri-o://ebb8437912a9e511dc06cdbc8ddfcc72ba2f54ab2250380cf20367dc92b51b9d" gracePeriod=30
Oct 30 22:39:33.365037 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:33.365005    5678 generic.go:356] "Generic (PLEG): container finished" podID="0e40fbc5-668a-4917-9344-dea69c625d26" containerID="ebb8437912a9e511dc06cdbc8ddfcc72ba2f54ab2250380cf20367dc92b51b9d" exitCode=0
Oct 30 22:39:33.365201 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:33.365052    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-75fb62-845c55c77b-gmd6f" event={"ID":"0e40fbc5-668a-4917-9344-dea69c625d26","Type":"ContainerDied","Data":"ebb8437912a9e511dc06cdbc8ddfcc72ba2f54ab2250380cf20367dc92b51b9d"}
Oct 30 22:39:33.433907 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:33.433851    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-75fb62-845c55c77b-gmd6f"
Oct 30 22:39:33.507355 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:33.507322    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e40fbc5-668a-4917-9344-dea69c625d26-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e40fbc5-668a-4917-9344-dea69c625d26-openshift-service-ca\") pod \"0e40fbc5-668a-4917-9344-dea69c625d26\" (UID: \"0e40fbc5-668a-4917-9344-dea69c625d26\") "
Oct 30 22:39:33.507508 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:33.507388    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e40fbc5-668a-4917-9344-dea69c625d26-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e40fbc5-668a-4917-9344-dea69c625d26-maas-api-tls\") pod \"0e40fbc5-668a-4917-9344-dea69c625d26\" (UID: \"0e40fbc5-668a-4917-9344-dea69c625d26\") "
Oct 30 22:39:33.507508 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:33.507456    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e40fbc5-668a-4917-9344-dea69c625d26-kube-api-access-bwb64\" (UniqueName: \"kubernetes.io/projected/0e40fbc5-668a-4917-9344-dea69c625d26-kube-api-access-bwb64\") pod \"0e40fbc5-668a-4917-9344-dea69c625d26\" (UID: \"0e40fbc5-668a-4917-9344-dea69c625d26\") "
Oct 30 22:39:33.507616 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:33.507599    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e40fbc5-668a-4917-9344-dea69c625d26-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e40fbc5-668a-4917-9344-dea69c625d26-metrics-certs\") pod \"0e40fbc5-668a-4917-9344-dea69c625d26\" (UID: \"0e40fbc5-668a-4917-9344-dea69c625d26\") "
Oct 30 22:39:33.507730 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:33.507708    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e40fbc5-668a-4917-9344-dea69c625d26-openshift-service-ca" pod "0e40fbc5-668a-4917-9344-dea69c625d26" (UID: "0e40fbc5-668a-4917-9344-dea69c625d26"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:39:33.507931 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:33.507917    5678 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e40fbc5-668a-4917-9344-dea69c625d26-openshift-service-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:39:33.509540 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:33.509516    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e40fbc5-668a-4917-9344-dea69c625d26-maas-api-tls" pod "0e40fbc5-668a-4917-9344-dea69c625d26" (UID: "0e40fbc5-668a-4917-9344-dea69c625d26"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:39:33.509540 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:33.509530    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e40fbc5-668a-4917-9344-dea69c625d26-metrics-certs" pod "0e40fbc5-668a-4917-9344-dea69c625d26" (UID: "0e40fbc5-668a-4917-9344-dea69c625d26"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:39:33.509691 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:33.509591    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e40fbc5-668a-4917-9344-dea69c625d26-kube-api-access-bwb64" pod "0e40fbc5-668a-4917-9344-dea69c625d26" (UID: "0e40fbc5-668a-4917-9344-dea69c625d26"). InnerVolumeSpecName "kube-api-access-bwb64". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:39:33.608828 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:33.608790    5678 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e40fbc5-668a-4917-9344-dea69c625d26-metrics-certs\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:39:33.608828 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:33.608816    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e40fbc5-668a-4917-9344-dea69c625d26-maas-api-tls\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:39:33.608828 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:33.608828    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwb64\" (UniqueName: \"kubernetes.io/projected/0e40fbc5-668a-4917-9344-dea69c625d26-kube-api-access-bwb64\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:39:34.372350 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:34.372320    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-75fb62-845c55c77b-gmd6f"
Oct 30 22:39:34.372826 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:34.372316    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-75fb62-845c55c77b-gmd6f" event={"ID":"0e40fbc5-668a-4917-9344-dea69c625d26","Type":"ContainerDied","Data":"fbb4a69e0edfb3965283f031a5bc887d08bf5e3cd9c80912801b985ad6ced1f7"}
Oct 30 22:39:34.372826 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:34.372473    5678 scope.go:122] "RemoveContainer" containerID="ebb8437912a9e511dc06cdbc8ddfcc72ba2f54ab2250380cf20367dc92b51b9d"
Oct 30 22:39:34.529759 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:34.529730    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-75fb62-845c55c77b-gmd6f"]
Oct 30 22:39:34.576750 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:34.576722    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-75fb62-845c55c77b-gmd6f"]
Oct 30 22:39:35.829308 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:35.829268    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-75fb62-5649854489-92pfg" podUID="95ef4304-6677-4325-8396-31575317404c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:39:35.829308 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:35.829298    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-75fb62-57ff7485f9-fl2h5" podUID="01b7d4f9-77ed-4292-a572-3a910bc8521e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:39:36.522091 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:36.522058    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e40fbc5-668a-4917-9344-dea69c625d26" path="/var/lib/kubelet/pods/0e40fbc5-668a-4917-9344-dea69c625d26/volumes"
Oct 30 22:39:36.843415 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:36.843378    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-506964-openshift-default-5bfdbc5849-4qwwq"
Oct 30 22:39:36.941564 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:36.941536    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64a0ba20-2130-4df1-b147-28a52a7d0e62-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64a0ba20-2130-4df1-b147-28a52a7d0e62-istio-envoy\") pod \"64a0ba20-2130-4df1-b147-28a52a7d0e62\" (UID: \"64a0ba20-2130-4df1-b147-28a52a7d0e62\") "
Oct 30 22:39:36.941692 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:36.941577    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64a0ba20-2130-4df1-b147-28a52a7d0e62-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64a0ba20-2130-4df1-b147-28a52a7d0e62-credential-socket\") pod \"64a0ba20-2130-4df1-b147-28a52a7d0e62\" (UID: \"64a0ba20-2130-4df1-b147-28a52a7d0e62\") "
Oct 30 22:39:36.941692 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:36.941624    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64a0ba20-2130-4df1-b147-28a52a7d0e62-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64a0ba20-2130-4df1-b147-28a52a7d0e62-istio-data\") pod \"64a0ba20-2130-4df1-b147-28a52a7d0e62\" (UID: \"64a0ba20-2130-4df1-b147-28a52a7d0e62\") "
Oct 30 22:39:36.941783 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:36.941714    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64a0ba20-2130-4df1-b147-28a52a7d0e62-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64a0ba20-2130-4df1-b147-28a52a7d0e62-workload-socket\") pod \"64a0ba20-2130-4df1-b147-28a52a7d0e62\" (UID: \"64a0ba20-2130-4df1-b147-28a52a7d0e62\") "
Oct 30 22:39:36.941826 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:36.941808    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64a0ba20-2130-4df1-b147-28a52a7d0e62-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64a0ba20-2130-4df1-b147-28a52a7d0e62-workload-certs\") pod \"64a0ba20-2130-4df1-b147-28a52a7d0e62\" (UID: \"64a0ba20-2130-4df1-b147-28a52a7d0e62\") "
Oct 30 22:39:36.941890 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:36.941871    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64a0ba20-2130-4df1-b147-28a52a7d0e62-kube-api-access-ktbzp\" (UniqueName: \"kubernetes.io/projected/64a0ba20-2130-4df1-b147-28a52a7d0e62-kube-api-access-ktbzp\") pod \"64a0ba20-2130-4df1-b147-28a52a7d0e62\" (UID: \"64a0ba20-2130-4df1-b147-28a52a7d0e62\") "
Oct 30 22:39:36.941964 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:36.941880    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64a0ba20-2130-4df1-b147-28a52a7d0e62-credential-socket" pod "64a0ba20-2130-4df1-b147-28a52a7d0e62" (UID: "64a0ba20-2130-4df1-b147-28a52a7d0e62"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:39:36.941964 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:36.941914    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64a0ba20-2130-4df1-b147-28a52a7d0e62-istio-token\" (UniqueName: \"kubernetes.io/projected/64a0ba20-2130-4df1-b147-28a52a7d0e62-istio-token\") pod \"64a0ba20-2130-4df1-b147-28a52a7d0e62\" (UID: \"64a0ba20-2130-4df1-b147-28a52a7d0e62\") "
Oct 30 22:39:36.941964 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:36.941946    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64a0ba20-2130-4df1-b147-28a52a7d0e62-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64a0ba20-2130-4df1-b147-28a52a7d0e62-istiod-ca-cert\") pod \"64a0ba20-2130-4df1-b147-28a52a7d0e62\" (UID: \"64a0ba20-2130-4df1-b147-28a52a7d0e62\") "
Oct 30 22:39:36.942143 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:36.941977    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/64a0ba20-2130-4df1-b147-28a52a7d0e62-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64a0ba20-2130-4df1-b147-28a52a7d0e62-istio-podinfo\") pod \"64a0ba20-2130-4df1-b147-28a52a7d0e62\" (UID: \"64a0ba20-2130-4df1-b147-28a52a7d0e62\") "
Oct 30 22:39:36.942143 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:36.941955    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64a0ba20-2130-4df1-b147-28a52a7d0e62-istio-data" pod "64a0ba20-2130-4df1-b147-28a52a7d0e62" (UID: "64a0ba20-2130-4df1-b147-28a52a7d0e62"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:39:36.942143 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:36.941990    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64a0ba20-2130-4df1-b147-28a52a7d0e62-workload-socket" pod "64a0ba20-2130-4df1-b147-28a52a7d0e62" (UID: "64a0ba20-2130-4df1-b147-28a52a7d0e62"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:39:36.942143 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:36.942067    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64a0ba20-2130-4df1-b147-28a52a7d0e62-workload-certs" pod "64a0ba20-2130-4df1-b147-28a52a7d0e62" (UID: "64a0ba20-2130-4df1-b147-28a52a7d0e62"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:39:36.942368 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:36.942286    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64a0ba20-2130-4df1-b147-28a52a7d0e62-istiod-ca-cert" pod "64a0ba20-2130-4df1-b147-28a52a7d0e62" (UID: "64a0ba20-2130-4df1-b147-28a52a7d0e62"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:39:36.942463 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:36.942441    5678 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64a0ba20-2130-4df1-b147-28a52a7d0e62-workload-socket\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:39:36.942561 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:36.942467    5678 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64a0ba20-2130-4df1-b147-28a52a7d0e62-workload-certs\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:39:36.942561 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:36.942481    5678 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64a0ba20-2130-4df1-b147-28a52a7d0e62-istiod-ca-cert\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:39:36.942561 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:36.942494    5678 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64a0ba20-2130-4df1-b147-28a52a7d0e62-credential-socket\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:39:36.942561 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:36.942508    5678 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64a0ba20-2130-4df1-b147-28a52a7d0e62-istio-data\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:39:36.944015 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:36.943988    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64a0ba20-2130-4df1-b147-28a52a7d0e62-istio-token" pod "64a0ba20-2130-4df1-b147-28a52a7d0e62" (UID: "64a0ba20-2130-4df1-b147-28a52a7d0e62"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:39:36.944110 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:36.944079    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/64a0ba20-2130-4df1-b147-28a52a7d0e62-istio-podinfo" pod "64a0ba20-2130-4df1-b147-28a52a7d0e62" (UID: "64a0ba20-2130-4df1-b147-28a52a7d0e62"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:39:36.944151 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:36.944130    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64a0ba20-2130-4df1-b147-28a52a7d0e62-istio-envoy" pod "64a0ba20-2130-4df1-b147-28a52a7d0e62" (UID: "64a0ba20-2130-4df1-b147-28a52a7d0e62"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:39:36.944288 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:36.944272    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64a0ba20-2130-4df1-b147-28a52a7d0e62-kube-api-access-ktbzp" pod "64a0ba20-2130-4df1-b147-28a52a7d0e62" (UID: "64a0ba20-2130-4df1-b147-28a52a7d0e62"). InnerVolumeSpecName "kube-api-access-ktbzp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:39:37.043044 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:37.043015    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktbzp\" (UniqueName: \"kubernetes.io/projected/64a0ba20-2130-4df1-b147-28a52a7d0e62-kube-api-access-ktbzp\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:39:37.043044 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:37.043040    5678 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64a0ba20-2130-4df1-b147-28a52a7d0e62-istio-token\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:39:37.043044 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:37.043051    5678 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64a0ba20-2130-4df1-b147-28a52a7d0e62-istio-podinfo\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:39:37.043347 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:37.043059    5678 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64a0ba20-2130-4df1-b147-28a52a7d0e62-istio-envoy\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:39:37.390704 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:37.390665    5678 generic.go:356] "Generic (PLEG): container finished" podID="64a0ba20-2130-4df1-b147-28a52a7d0e62" containerID="2a2a0c368ccd60107ea46ad21cd211e2bdf53734db49f376966c0db909f47426" exitCode=0
Oct 30 22:39:37.390878 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:37.390712    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-506964-openshift-default-5bfdbc5849-4qwwq"
Oct 30 22:39:37.390878 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:37.390749    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-506964-openshift-default-5bfdbc5849-4qwwq" event={"ID":"64a0ba20-2130-4df1-b147-28a52a7d0e62","Type":"ContainerDied","Data":"2a2a0c368ccd60107ea46ad21cd211e2bdf53734db49f376966c0db909f47426"}
Oct 30 22:39:37.390878 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:37.390828    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-506964-openshift-default-5bfdbc5849-4qwwq" event={"ID":"64a0ba20-2130-4df1-b147-28a52a7d0e62","Type":"ContainerDied","Data":"5fa835ed7ad27e0b73b0fdd0e7d3886130505dd80175fdde672583c9e89563b2"}
Oct 30 22:39:37.390878 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:37.390850    5678 scope.go:122] "RemoveContainer" containerID="2a2a0c368ccd60107ea46ad21cd211e2bdf53734db49f376966c0db909f47426"
Oct 30 22:39:37.400558 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:37.400538    5678 scope.go:122] "RemoveContainer" containerID="03a0781f38eb9c11adf9a9b59c0598959af5afd188736fd885086d08384bc326"
Oct 30 22:39:37.409169 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:37.409152    5678 scope.go:122] "RemoveContainer" containerID="2a2a0c368ccd60107ea46ad21cd211e2bdf53734db49f376966c0db909f47426"
Oct 30 22:39:37.409444 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:39:37.409418    5678 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a2a0c368ccd60107ea46ad21cd211e2bdf53734db49f376966c0db909f47426\": container with ID starting with 2a2a0c368ccd60107ea46ad21cd211e2bdf53734db49f376966c0db909f47426 not found: ID does not exist" containerID="2a2a0c368ccd60107ea46ad21cd211e2bdf53734db49f376966c0db909f47426"
Oct 30 22:39:37.409517 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:37.409453    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a2a0c368ccd60107ea46ad21cd211e2bdf53734db49f376966c0db909f47426"} err="failed to get container status \"2a2a0c368ccd60107ea46ad21cd211e2bdf53734db49f376966c0db909f47426\": rpc error: code = NotFound desc = could not find container \"2a2a0c368ccd60107ea46ad21cd211e2bdf53734db49f376966c0db909f47426\": container with ID starting with 2a2a0c368ccd60107ea46ad21cd211e2bdf53734db49f376966c0db909f47426 not found: ID does not exist"
Oct 30 22:39:37.409517 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:37.409473    5678 scope.go:122] "RemoveContainer" containerID="03a0781f38eb9c11adf9a9b59c0598959af5afd188736fd885086d08384bc326"
Oct 30 22:39:37.409732 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:39:37.409713    5678 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03a0781f38eb9c11adf9a9b59c0598959af5afd188736fd885086d08384bc326\": container with ID starting with 03a0781f38eb9c11adf9a9b59c0598959af5afd188736fd885086d08384bc326 not found: ID does not exist" containerID="03a0781f38eb9c11adf9a9b59c0598959af5afd188736fd885086d08384bc326"
Oct 30 22:39:37.409782 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:37.409738    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03a0781f38eb9c11adf9a9b59c0598959af5afd188736fd885086d08384bc326"} err="failed to get container status \"03a0781f38eb9c11adf9a9b59c0598959af5afd188736fd885086d08384bc326\": rpc error: code = NotFound desc = could not find container \"03a0781f38eb9c11adf9a9b59c0598959af5afd188736fd885086d08384bc326\": container with ID starting with 03a0781f38eb9c11adf9a9b59c0598959af5afd188736fd885086d08384bc326 not found: ID does not exist"
Oct 30 22:39:37.576220 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:37.576187    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-506964-openshift-default-5bfdbc5849-4qwwq"]
Oct 30 22:39:37.623207 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:37.623176    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-506964-openshift-default-5bfdbc5849-4qwwq"]
Oct 30 22:39:38.522306 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:38.522262    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64a0ba20-2130-4df1-b147-28a52a7d0e62" path="/var/lib/kubelet/pods/64a0ba20-2130-4df1-b147-28a52a7d0e62/volumes"
Oct 30 22:39:40.571308 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:39:40.571273    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 22:39:42.558264 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:42.558228    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-75fb62-openshift-default-c9cc6b459-4f2fg"]
Oct 30 22:39:42.558865 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:42.558821    5678 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-75fb62-openshift-default-c9cc6b459-4f2fg" podUID="f1691924-5354-4690-8709-6a20bfd81a7e" containerName="istio-proxy" containerID="cri-o://57eb1b9435314b42a559b796b8611709458066c0d6fe43a95e34c29d1fa6165f" gracePeriod=30
Oct 30 22:39:42.637423 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:42.625560    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-75fb62-5649854489-92pfg"
Oct 30 22:39:42.690040 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:42.690017    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-75fb62-57ff7485f9-fl2h5"
Oct 30 22:39:42.804048 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:42.804019    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01b7d4f9-77ed-4292-a572-3a910bc8521e-kube-api-access-26pd4\" (UniqueName: \"kubernetes.io/projected/01b7d4f9-77ed-4292-a572-3a910bc8521e-kube-api-access-26pd4\") pod \"01b7d4f9-77ed-4292-a572-3a910bc8521e\" (UID: \"01b7d4f9-77ed-4292-a572-3a910bc8521e\") "
Oct 30 22:39:42.804210 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:42.804151    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95ef4304-6677-4325-8396-31575317404c-kube-api-access-d5g96\" (UniqueName: \"kubernetes.io/projected/95ef4304-6677-4325-8396-31575317404c-kube-api-access-d5g96\") pod \"95ef4304-6677-4325-8396-31575317404c\" (UID: \"95ef4304-6677-4325-8396-31575317404c\") "
Oct 30 22:39:42.804210 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:42.804175    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01b7d4f9-77ed-4292-a572-3a910bc8521e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01b7d4f9-77ed-4292-a572-3a910bc8521e-plugins-config-volume\") pod \"01b7d4f9-77ed-4292-a572-3a910bc8521e\" (UID: \"01b7d4f9-77ed-4292-a572-3a910bc8521e\") "
Oct 30 22:39:42.804282 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:42.804239    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95ef4304-6677-4325-8396-31575317404c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95ef4304-6677-4325-8396-31575317404c-plugins-config-volume\") pod \"95ef4304-6677-4325-8396-31575317404c\" (UID: \"95ef4304-6677-4325-8396-31575317404c\") "
Oct 30 22:39:42.804688 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:42.804660    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01b7d4f9-77ed-4292-a572-3a910bc8521e-plugins-config-volume" pod "01b7d4f9-77ed-4292-a572-3a910bc8521e" (UID: "01b7d4f9-77ed-4292-a572-3a910bc8521e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:39:42.804838 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:42.804677    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95ef4304-6677-4325-8396-31575317404c-plugins-config-volume" pod "95ef4304-6677-4325-8396-31575317404c" (UID: "95ef4304-6677-4325-8396-31575317404c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:39:42.806537 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:42.806421    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01b7d4f9-77ed-4292-a572-3a910bc8521e-kube-api-access-26pd4" pod "01b7d4f9-77ed-4292-a572-3a910bc8521e" (UID: "01b7d4f9-77ed-4292-a572-3a910bc8521e"). InnerVolumeSpecName "kube-api-access-26pd4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:39:42.806537 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:42.806457    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95ef4304-6677-4325-8396-31575317404c-kube-api-access-d5g96" pod "95ef4304-6677-4325-8396-31575317404c" (UID: "95ef4304-6677-4325-8396-31575317404c"). InnerVolumeSpecName "kube-api-access-d5g96". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:39:42.905822 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:42.905787    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26pd4\" (UniqueName: \"kubernetes.io/projected/01b7d4f9-77ed-4292-a572-3a910bc8521e-kube-api-access-26pd4\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:39:42.905822 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:42.905815    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5g96\" (UniqueName: \"kubernetes.io/projected/95ef4304-6677-4325-8396-31575317404c-kube-api-access-d5g96\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:39:42.905822 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:42.905825    5678 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01b7d4f9-77ed-4292-a572-3a910bc8521e-plugins-config-volume\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:39:42.906067 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:42.905834    5678 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95ef4304-6677-4325-8396-31575317404c-plugins-config-volume\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:39:43.427140 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:43.427107    5678 generic.go:356] "Generic (PLEG): container finished" podID="01b7d4f9-77ed-4292-a572-3a910bc8521e" containerID="708b64baddc2018a602c41d286ea7bb816b507bc0119e5f035dd3bafa3d7336a" exitCode=0
Oct 30 22:39:43.427329 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:43.427154    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-75fb62-57ff7485f9-fl2h5"
Oct 30 22:39:43.427329 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:43.427193    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-75fb62-57ff7485f9-fl2h5" event={"ID":"01b7d4f9-77ed-4292-a572-3a910bc8521e","Type":"ContainerDied","Data":"708b64baddc2018a602c41d286ea7bb816b507bc0119e5f035dd3bafa3d7336a"}
Oct 30 22:39:43.427329 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:43.427243    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-75fb62-57ff7485f9-fl2h5" event={"ID":"01b7d4f9-77ed-4292-a572-3a910bc8521e","Type":"ContainerDied","Data":"8701985c82dcac1db69edfaeb7c183b66c73dc94baa5004f13854a03d6f88a0d"}
Oct 30 22:39:43.427329 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:43.427264    5678 scope.go:122] "RemoveContainer" containerID="708b64baddc2018a602c41d286ea7bb816b507bc0119e5f035dd3bafa3d7336a"
Oct 30 22:39:43.428551 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:43.428529    5678 generic.go:356] "Generic (PLEG): container finished" podID="95ef4304-6677-4325-8396-31575317404c" containerID="e04707050b9f01b8187bff71c74d10dcfd51ffa90ff37773891da0a9e02b6c7e" exitCode=0
Oct 30 22:39:43.428723 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:43.428572    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-75fb62-5649854489-92pfg"
Oct 30 22:39:43.428723 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:43.428609    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-75fb62-5649854489-92pfg" event={"ID":"95ef4304-6677-4325-8396-31575317404c","Type":"ContainerDied","Data":"e04707050b9f01b8187bff71c74d10dcfd51ffa90ff37773891da0a9e02b6c7e"}
Oct 30 22:39:43.428723 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:43.428654    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-75fb62-5649854489-92pfg" event={"ID":"95ef4304-6677-4325-8396-31575317404c","Type":"ContainerDied","Data":"b2c9bafd1a27d5cb74e5929a63a88b7f216a347be9df0d7c232aa8aa9d123af4"}
Oct 30 22:39:43.438275 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:43.438247    5678 scope.go:122] "RemoveContainer" containerID="708b64baddc2018a602c41d286ea7bb816b507bc0119e5f035dd3bafa3d7336a"
Oct 30 22:39:43.438579 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:39:43.438551    5678 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"708b64baddc2018a602c41d286ea7bb816b507bc0119e5f035dd3bafa3d7336a\": container with ID starting with 708b64baddc2018a602c41d286ea7bb816b507bc0119e5f035dd3bafa3d7336a not found: ID does not exist" containerID="708b64baddc2018a602c41d286ea7bb816b507bc0119e5f035dd3bafa3d7336a"
Oct 30 22:39:43.438822 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:43.438590    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"708b64baddc2018a602c41d286ea7bb816b507bc0119e5f035dd3bafa3d7336a"} err="failed to get container status \"708b64baddc2018a602c41d286ea7bb816b507bc0119e5f035dd3bafa3d7336a\": rpc error: code = NotFound desc = could not find container \"708b64baddc2018a602c41d286ea7bb816b507bc0119e5f035dd3bafa3d7336a\": container with ID starting with 708b64baddc2018a602c41d286ea7bb816b507bc0119e5f035dd3bafa3d7336a not found: ID does not exist"
Oct 30 22:39:43.438822 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:43.438616    5678 scope.go:122] "RemoveContainer" containerID="e04707050b9f01b8187bff71c74d10dcfd51ffa90ff37773891da0a9e02b6c7e"
Oct 30 22:39:43.447003 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:43.446840    5678 scope.go:122] "RemoveContainer" containerID="e04707050b9f01b8187bff71c74d10dcfd51ffa90ff37773891da0a9e02b6c7e"
Oct 30 22:39:43.447115 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:39:43.447085    5678 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e04707050b9f01b8187bff71c74d10dcfd51ffa90ff37773891da0a9e02b6c7e\": container with ID starting with e04707050b9f01b8187bff71c74d10dcfd51ffa90ff37773891da0a9e02b6c7e not found: ID does not exist" containerID="e04707050b9f01b8187bff71c74d10dcfd51ffa90ff37773891da0a9e02b6c7e"
Oct 30 22:39:43.447157 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:43.447121    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e04707050b9f01b8187bff71c74d10dcfd51ffa90ff37773891da0a9e02b6c7e"} err="failed to get container status \"e04707050b9f01b8187bff71c74d10dcfd51ffa90ff37773891da0a9e02b6c7e\": rpc error: code = NotFound desc = could not find container \"e04707050b9f01b8187bff71c74d10dcfd51ffa90ff37773891da0a9e02b6c7e\": container with ID starting with e04707050b9f01b8187bff71c74d10dcfd51ffa90ff37773891da0a9e02b6c7e not found: ID does not exist"
Oct 30 22:39:43.584266 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:43.584236    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-75fb62-57ff7485f9-fl2h5"]
Oct 30 22:39:43.588307 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:43.588282    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-75fb62-57ff7485f9-fl2h5"]
Oct 30 22:39:43.691998 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:43.691961    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-75fb62-5649854489-92pfg"]
Oct 30 22:39:43.696974 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:43.696951    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-75fb62-5649854489-92pfg"]
Oct 30 22:39:44.522164 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:44.522131    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01b7d4f9-77ed-4292-a572-3a910bc8521e" path="/var/lib/kubelet/pods/01b7d4f9-77ed-4292-a572-3a910bc8521e/volumes"
Oct 30 22:39:44.522534 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:44.522519    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95ef4304-6677-4325-8396-31575317404c" path="/var/lib/kubelet/pods/95ef4304-6677-4325-8396-31575317404c/volumes"
Oct 30 22:39:47.815312 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:47.815288    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-75fb62-openshift-default-c9cc6b459-4f2fg"
Oct 30 22:39:47.956654 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:47.956624    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1691924-5354-4690-8709-6a20bfd81a7e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1691924-5354-4690-8709-6a20bfd81a7e-istio-envoy\") pod \"f1691924-5354-4690-8709-6a20bfd81a7e\" (UID: \"f1691924-5354-4690-8709-6a20bfd81a7e\") "
Oct 30 22:39:47.956654 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:47.956659    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1691924-5354-4690-8709-6a20bfd81a7e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1691924-5354-4690-8709-6a20bfd81a7e-workload-certs\") pod \"f1691924-5354-4690-8709-6a20bfd81a7e\" (UID: \"f1691924-5354-4690-8709-6a20bfd81a7e\") "
Oct 30 22:39:47.956963 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:47.956683    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1691924-5354-4690-8709-6a20bfd81a7e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1691924-5354-4690-8709-6a20bfd81a7e-istio-data\") pod \"f1691924-5354-4690-8709-6a20bfd81a7e\" (UID: \"f1691924-5354-4690-8709-6a20bfd81a7e\") "
Oct 30 22:39:47.956963 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:47.956723    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1691924-5354-4690-8709-6a20bfd81a7e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1691924-5354-4690-8709-6a20bfd81a7e-istiod-ca-cert\") pod \"f1691924-5354-4690-8709-6a20bfd81a7e\" (UID: \"f1691924-5354-4690-8709-6a20bfd81a7e\") "
Oct 30 22:39:47.956963 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:47.956753    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1691924-5354-4690-8709-6a20bfd81a7e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1691924-5354-4690-8709-6a20bfd81a7e-credential-socket\") pod \"f1691924-5354-4690-8709-6a20bfd81a7e\" (UID: \"f1691924-5354-4690-8709-6a20bfd81a7e\") "
Oct 30 22:39:47.956963 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:47.956781    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f1691924-5354-4690-8709-6a20bfd81a7e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1691924-5354-4690-8709-6a20bfd81a7e-istio-podinfo\") pod \"f1691924-5354-4690-8709-6a20bfd81a7e\" (UID: \"f1691924-5354-4690-8709-6a20bfd81a7e\") "
Oct 30 22:39:47.956963 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:47.956818    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1691924-5354-4690-8709-6a20bfd81a7e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1691924-5354-4690-8709-6a20bfd81a7e-workload-socket\") pod \"f1691924-5354-4690-8709-6a20bfd81a7e\" (UID: \"f1691924-5354-4690-8709-6a20bfd81a7e\") "
Oct 30 22:39:47.956963 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:47.956855    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1691924-5354-4690-8709-6a20bfd81a7e-kube-api-access-wqwgx\" (UniqueName: \"kubernetes.io/projected/f1691924-5354-4690-8709-6a20bfd81a7e-kube-api-access-wqwgx\") pod \"f1691924-5354-4690-8709-6a20bfd81a7e\" (UID: \"f1691924-5354-4690-8709-6a20bfd81a7e\") "
Oct 30 22:39:47.956963 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:47.956901    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1691924-5354-4690-8709-6a20bfd81a7e-istio-token\" (UniqueName: \"kubernetes.io/projected/f1691924-5354-4690-8709-6a20bfd81a7e-istio-token\") pod \"f1691924-5354-4690-8709-6a20bfd81a7e\" (UID: \"f1691924-5354-4690-8709-6a20bfd81a7e\") "
Oct 30 22:39:47.956963 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:47.956952    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1691924-5354-4690-8709-6a20bfd81a7e-workload-certs" pod "f1691924-5354-4690-8709-6a20bfd81a7e" (UID: "f1691924-5354-4690-8709-6a20bfd81a7e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:39:47.957386 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:47.956981    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1691924-5354-4690-8709-6a20bfd81a7e-istio-data" pod "f1691924-5354-4690-8709-6a20bfd81a7e" (UID: "f1691924-5354-4690-8709-6a20bfd81a7e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:39:47.957386 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:47.957100    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1691924-5354-4690-8709-6a20bfd81a7e-workload-socket" pod "f1691924-5354-4690-8709-6a20bfd81a7e" (UID: "f1691924-5354-4690-8709-6a20bfd81a7e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:39:47.957386 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:47.957201    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1691924-5354-4690-8709-6a20bfd81a7e-istiod-ca-cert" pod "f1691924-5354-4690-8709-6a20bfd81a7e" (UID: "f1691924-5354-4690-8709-6a20bfd81a7e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:39:47.957386 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:47.957349    5678 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1691924-5354-4690-8709-6a20bfd81a7e-workload-certs\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:39:47.957386 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:47.957371    5678 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1691924-5354-4690-8709-6a20bfd81a7e-istio-data\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:39:47.957386 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:47.957388    5678 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1691924-5354-4690-8709-6a20bfd81a7e-istiod-ca-cert\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:39:47.957725 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:47.957419    5678 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1691924-5354-4690-8709-6a20bfd81a7e-workload-socket\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:39:47.957725 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:47.957641    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1691924-5354-4690-8709-6a20bfd81a7e-credential-socket" pod "f1691924-5354-4690-8709-6a20bfd81a7e" (UID: "f1691924-5354-4690-8709-6a20bfd81a7e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:39:47.958878 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:47.958852    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1691924-5354-4690-8709-6a20bfd81a7e-istio-envoy" pod "f1691924-5354-4690-8709-6a20bfd81a7e" (UID: "f1691924-5354-4690-8709-6a20bfd81a7e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:39:47.959068 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:47.959051    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f1691924-5354-4690-8709-6a20bfd81a7e-istio-podinfo" pod "f1691924-5354-4690-8709-6a20bfd81a7e" (UID: "f1691924-5354-4690-8709-6a20bfd81a7e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:39:47.959068 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:47.959056    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1691924-5354-4690-8709-6a20bfd81a7e-kube-api-access-wqwgx" pod "f1691924-5354-4690-8709-6a20bfd81a7e" (UID: "f1691924-5354-4690-8709-6a20bfd81a7e"). InnerVolumeSpecName "kube-api-access-wqwgx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:39:47.959156 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:47.959141    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1691924-5354-4690-8709-6a20bfd81a7e-istio-token" pod "f1691924-5354-4690-8709-6a20bfd81a7e" (UID: "f1691924-5354-4690-8709-6a20bfd81a7e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:39:48.058878 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:48.058839    5678 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1691924-5354-4690-8709-6a20bfd81a7e-credential-socket\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:39:48.058878 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:48.058870    5678 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1691924-5354-4690-8709-6a20bfd81a7e-istio-podinfo\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:39:48.059114 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:48.058885    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqwgx\" (UniqueName: \"kubernetes.io/projected/f1691924-5354-4690-8709-6a20bfd81a7e-kube-api-access-wqwgx\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:39:48.059114 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:48.058900    5678 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1691924-5354-4690-8709-6a20bfd81a7e-istio-token\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:39:48.059114 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:48.058912    5678 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1691924-5354-4690-8709-6a20bfd81a7e-istio-envoy\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:39:48.461248 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:48.461209    5678 generic.go:356] "Generic (PLEG): container finished" podID="f1691924-5354-4690-8709-6a20bfd81a7e" containerID="57eb1b9435314b42a559b796b8611709458066c0d6fe43a95e34c29d1fa6165f" exitCode=0
Oct 30 22:39:48.461463 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:48.461267    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-75fb62-openshift-default-c9cc6b459-4f2fg"
Oct 30 22:39:48.461463 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:48.461287    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-75fb62-openshift-default-c9cc6b459-4f2fg" event={"ID":"f1691924-5354-4690-8709-6a20bfd81a7e","Type":"ContainerDied","Data":"57eb1b9435314b42a559b796b8611709458066c0d6fe43a95e34c29d1fa6165f"}
Oct 30 22:39:48.461463 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:48.461355    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-75fb62-openshift-default-c9cc6b459-4f2fg" event={"ID":"f1691924-5354-4690-8709-6a20bfd81a7e","Type":"ContainerDied","Data":"53287d1705251e672c8b74eb6e3d3e046bed50ee982198749923cb37dd4e2b71"}
Oct 30 22:39:48.461463 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:48.461377    5678 scope.go:122] "RemoveContainer" containerID="57eb1b9435314b42a559b796b8611709458066c0d6fe43a95e34c29d1fa6165f"
Oct 30 22:39:48.471201 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:48.471184    5678 scope.go:122] "RemoveContainer" containerID="df99e50604825f2460a311054363d6eeaafae354297ddbb74bce51ae632392a6"
Oct 30 22:39:48.479335 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:48.479316    5678 scope.go:122] "RemoveContainer" containerID="57eb1b9435314b42a559b796b8611709458066c0d6fe43a95e34c29d1fa6165f"
Oct 30 22:39:48.479618 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:39:48.479600    5678 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57eb1b9435314b42a559b796b8611709458066c0d6fe43a95e34c29d1fa6165f\": container with ID starting with 57eb1b9435314b42a559b796b8611709458066c0d6fe43a95e34c29d1fa6165f not found: ID does not exist" containerID="57eb1b9435314b42a559b796b8611709458066c0d6fe43a95e34c29d1fa6165f"
Oct 30 22:39:48.479700 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:48.479634    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57eb1b9435314b42a559b796b8611709458066c0d6fe43a95e34c29d1fa6165f"} err="failed to get container status \"57eb1b9435314b42a559b796b8611709458066c0d6fe43a95e34c29d1fa6165f\": rpc error: code = NotFound desc = could not find container \"57eb1b9435314b42a559b796b8611709458066c0d6fe43a95e34c29d1fa6165f\": container with ID starting with 57eb1b9435314b42a559b796b8611709458066c0d6fe43a95e34c29d1fa6165f not found: ID does not exist"
Oct 30 22:39:48.479700 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:48.479661    5678 scope.go:122] "RemoveContainer" containerID="df99e50604825f2460a311054363d6eeaafae354297ddbb74bce51ae632392a6"
Oct 30 22:39:48.479947 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:39:48.479929    5678 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df99e50604825f2460a311054363d6eeaafae354297ddbb74bce51ae632392a6\": container with ID starting with df99e50604825f2460a311054363d6eeaafae354297ddbb74bce51ae632392a6 not found: ID does not exist" containerID="df99e50604825f2460a311054363d6eeaafae354297ddbb74bce51ae632392a6"
Oct 30 22:39:48.479994 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:48.479953    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df99e50604825f2460a311054363d6eeaafae354297ddbb74bce51ae632392a6"} err="failed to get container status \"df99e50604825f2460a311054363d6eeaafae354297ddbb74bce51ae632392a6\": rpc error: code = NotFound desc = could not find container \"df99e50604825f2460a311054363d6eeaafae354297ddbb74bce51ae632392a6\": container with ID starting with df99e50604825f2460a311054363d6eeaafae354297ddbb74bce51ae632392a6 not found: ID does not exist"
Oct 30 22:39:48.619999 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:48.619961    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-75fb62-openshift-default-c9cc6b459-4f2fg"]
Oct 30 22:39:48.666739 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:48.666702    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-75fb62-openshift-default-c9cc6b459-4f2fg"]
Oct 30 22:39:50.521774 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:50.521741    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1691924-5354-4690-8709-6a20bfd81a7e" path="/var/lib/kubelet/pods/f1691924-5354-4690-8709-6a20bfd81a7e/volumes"
Oct 30 22:39:53.821891 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:53.821856    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-07e9c8-5bd8cdfbff-h8vtx"]
Oct 30 22:39:53.922380 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:53.922350    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-07e9c8-5f45d5ff75-z224x"]
Oct 30 22:39:58.822833 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:58.822797    5678 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-07e9c8-5bd8cdfbff-h8vtx" podUID="ab8b95b5-8eca-405d-bfa9-195e5b4f68f7" containerName="payload-processing" containerID="cri-o://c23c11290c0bf58481911bdb94015c88696afc359ce77e5cdd6fa848dc598a3c" gracePeriod=25
Oct 30 22:39:58.923026 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:39:58.922970    5678 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-07e9c8-5f45d5ff75-z224x" podUID="47873f24-8e29-439b-8017-776ece323a1c" containerName="payload-pre-processing" containerID="cri-o://4fd044d0bf9c012d99f4d112c479b74f487d514963982abe3a65523687363140" gracePeriod=25
Oct 30 22:40:00.115072 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:00.115029    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-07e9c8-f548454dc-lsx4x"]
Oct 30 22:40:00.115600 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:00.115356    5678 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-07e9c8-f548454dc-lsx4x" podUID="c01c1a4e-7d4d-45ca-b92e-c1cdd9013059" containerName="maas-api" containerID="cri-o://86131e183696784ce2e01813bda8b728058837755e9e5fadd1379fa41bac8e99" gracePeriod=30
Oct 30 22:40:00.375537 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:00.375471    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-07e9c8-f548454dc-lsx4x"
Oct 30 22:40:00.469067 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:00.469028    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c01c1a4e-7d4d-45ca-b92e-c1cdd9013059-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c01c1a4e-7d4d-45ca-b92e-c1cdd9013059-metrics-certs\") pod \"c01c1a4e-7d4d-45ca-b92e-c1cdd9013059\" (UID: \"c01c1a4e-7d4d-45ca-b92e-c1cdd9013059\") "
Oct 30 22:40:00.469225 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:00.469094    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c01c1a4e-7d4d-45ca-b92e-c1cdd9013059-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c01c1a4e-7d4d-45ca-b92e-c1cdd9013059-maas-api-tls\") pod \"c01c1a4e-7d4d-45ca-b92e-c1cdd9013059\" (UID: \"c01c1a4e-7d4d-45ca-b92e-c1cdd9013059\") "
Oct 30 22:40:00.469225 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:00.469121    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c01c1a4e-7d4d-45ca-b92e-c1cdd9013059-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c01c1a4e-7d4d-45ca-b92e-c1cdd9013059-openshift-service-ca\") pod \"c01c1a4e-7d4d-45ca-b92e-c1cdd9013059\" (UID: \"c01c1a4e-7d4d-45ca-b92e-c1cdd9013059\") "
Oct 30 22:40:00.469225 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:00.469170    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c01c1a4e-7d4d-45ca-b92e-c1cdd9013059-kube-api-access-5rvs8\" (UniqueName: \"kubernetes.io/projected/c01c1a4e-7d4d-45ca-b92e-c1cdd9013059-kube-api-access-5rvs8\") pod \"c01c1a4e-7d4d-45ca-b92e-c1cdd9013059\" (UID: \"c01c1a4e-7d4d-45ca-b92e-c1cdd9013059\") "
Oct 30 22:40:00.469567 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:00.469540    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c01c1a4e-7d4d-45ca-b92e-c1cdd9013059-openshift-service-ca" pod "c01c1a4e-7d4d-45ca-b92e-c1cdd9013059" (UID: "c01c1a4e-7d4d-45ca-b92e-c1cdd9013059"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:40:00.471421 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:00.471368    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c01c1a4e-7d4d-45ca-b92e-c1cdd9013059-metrics-certs" pod "c01c1a4e-7d4d-45ca-b92e-c1cdd9013059" (UID: "c01c1a4e-7d4d-45ca-b92e-c1cdd9013059"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:40:00.471528 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:00.471431    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c01c1a4e-7d4d-45ca-b92e-c1cdd9013059-maas-api-tls" pod "c01c1a4e-7d4d-45ca-b92e-c1cdd9013059" (UID: "c01c1a4e-7d4d-45ca-b92e-c1cdd9013059"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:40:00.472456 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:00.472429    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c01c1a4e-7d4d-45ca-b92e-c1cdd9013059-kube-api-access-5rvs8" pod "c01c1a4e-7d4d-45ca-b92e-c1cdd9013059" (UID: "c01c1a4e-7d4d-45ca-b92e-c1cdd9013059"). InnerVolumeSpecName "kube-api-access-5rvs8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:40:00.528168 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:00.528133    5678 generic.go:356] "Generic (PLEG): container finished" podID="c01c1a4e-7d4d-45ca-b92e-c1cdd9013059" containerID="86131e183696784ce2e01813bda8b728058837755e9e5fadd1379fa41bac8e99" exitCode=0
Oct 30 22:40:00.528310 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:00.528178    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-07e9c8-f548454dc-lsx4x"
Oct 30 22:40:00.528310 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:00.528207    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-07e9c8-f548454dc-lsx4x" event={"ID":"c01c1a4e-7d4d-45ca-b92e-c1cdd9013059","Type":"ContainerDied","Data":"86131e183696784ce2e01813bda8b728058837755e9e5fadd1379fa41bac8e99"}
Oct 30 22:40:00.528310 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:00.528261    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-07e9c8-f548454dc-lsx4x" event={"ID":"c01c1a4e-7d4d-45ca-b92e-c1cdd9013059","Type":"ContainerDied","Data":"1409363a486fc91851091cf585ce4455d76961fdd2475860bc814c492f593d8f"}
Oct 30 22:40:00.528310 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:00.528284    5678 scope.go:122] "RemoveContainer" containerID="86131e183696784ce2e01813bda8b728058837755e9e5fadd1379fa41bac8e99"
Oct 30 22:40:00.538499 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:00.538479    5678 scope.go:122] "RemoveContainer" containerID="86131e183696784ce2e01813bda8b728058837755e9e5fadd1379fa41bac8e99"
Oct 30 22:40:00.538774 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:40:00.538758    5678 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86131e183696784ce2e01813bda8b728058837755e9e5fadd1379fa41bac8e99\": container with ID starting with 86131e183696784ce2e01813bda8b728058837755e9e5fadd1379fa41bac8e99 not found: ID does not exist" containerID="86131e183696784ce2e01813bda8b728058837755e9e5fadd1379fa41bac8e99"
Oct 30 22:40:00.538824 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:00.538781    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86131e183696784ce2e01813bda8b728058837755e9e5fadd1379fa41bac8e99"} err="failed to get container status \"86131e183696784ce2e01813bda8b728058837755e9e5fadd1379fa41bac8e99\": rpc error: code = NotFound desc = could not find container \"86131e183696784ce2e01813bda8b728058837755e9e5fadd1379fa41bac8e99\": container with ID starting with 86131e183696784ce2e01813bda8b728058837755e9e5fadd1379fa41bac8e99 not found: ID does not exist"
Oct 30 22:40:00.570893 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:00.570852    5678 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c01c1a4e-7d4d-45ca-b92e-c1cdd9013059-metrics-certs\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:00.570893 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:00.570882    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c01c1a4e-7d4d-45ca-b92e-c1cdd9013059-maas-api-tls\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:00.570893 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:00.570898    5678 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c01c1a4e-7d4d-45ca-b92e-c1cdd9013059-openshift-service-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:00.571107 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:00.570912    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rvs8\" (UniqueName: \"kubernetes.io/projected/c01c1a4e-7d4d-45ca-b92e-c1cdd9013059-kube-api-access-5rvs8\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:00.684679 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:00.684589    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-07e9c8-f548454dc-lsx4x"]
Oct 30 22:40:00.691491 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:00.691457    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-07e9c8-f548454dc-lsx4x"]
Oct 30 22:40:02.522857 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:02.522783    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c01c1a4e-7d4d-45ca-b92e-c1cdd9013059" path="/var/lib/kubelet/pods/c01c1a4e-7d4d-45ca-b92e-c1cdd9013059/volumes"
Oct 30 22:40:04.158331 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:04.158288    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-07e9c8-5f45d5ff75-z224x" podUID="47873f24-8e29-439b-8017-776ece323a1c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:40:04.158331 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:04.158310    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-07e9c8-5bd8cdfbff-h8vtx" podUID="ab8b95b5-8eca-405d-bfa9-195e5b4f68f7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:40:09.662795 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:09.662762    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-07e9c8-openshift-default-5944f97d49-m76c9"]
Oct 30 22:40:09.663217 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:09.662991    5678 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-07e9c8-openshift-default-5944f97d49-m76c9" podUID="f8eba014-2735-4189-a982-e33413ce8e4b" containerName="istio-proxy" containerID="cri-o://38afa7c023ddcdbab4c782283d3887435ad57de709f00e5888f0dcd9fd7e9995" gracePeriod=30
Oct 30 22:40:10.571779 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:40:10.571744    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 22:40:14.158278 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:14.158232    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-07e9c8-5bd8cdfbff-h8vtx" podUID="ab8b95b5-8eca-405d-bfa9-195e5b4f68f7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:40:14.158703 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:14.158250    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-07e9c8-5f45d5ff75-z224x" podUID="47873f24-8e29-439b-8017-776ece323a1c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:40:14.278525 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:14.278495    5678 patch_prober.go:29] interesting pod/e2e-inf-b-07e9c8-openshift-default-5944f97d49-m76c9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:40:14.278687 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:14.278542    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-07e9c8-openshift-default-5944f97d49-m76c9" podUID="f8eba014-2735-4189-a982-e33413ce8e4b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:40:14.912575 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:14.912550    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-07e9c8-openshift-default-5944f97d49-m76c9"
Oct 30 22:40:15.013769 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:15.013706    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8eba014-2735-4189-a982-e33413ce8e4b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8eba014-2735-4189-a982-e33413ce8e4b-credential-socket\") pod \"f8eba014-2735-4189-a982-e33413ce8e4b\" (UID: \"f8eba014-2735-4189-a982-e33413ce8e4b\") "
Oct 30 22:40:15.013769 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:15.013739    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8eba014-2735-4189-a982-e33413ce8e4b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8eba014-2735-4189-a982-e33413ce8e4b-workload-socket\") pod \"f8eba014-2735-4189-a982-e33413ce8e4b\" (UID: \"f8eba014-2735-4189-a982-e33413ce8e4b\") "
Oct 30 22:40:15.013769 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:15.013763    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8eba014-2735-4189-a982-e33413ce8e4b-istio-token\" (UniqueName: \"kubernetes.io/projected/f8eba014-2735-4189-a982-e33413ce8e4b-istio-token\") pod \"f8eba014-2735-4189-a982-e33413ce8e4b\" (UID: \"f8eba014-2735-4189-a982-e33413ce8e4b\") "
Oct 30 22:40:15.014041 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:15.013815    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8eba014-2735-4189-a982-e33413ce8e4b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8eba014-2735-4189-a982-e33413ce8e4b-workload-certs\") pod \"f8eba014-2735-4189-a982-e33413ce8e4b\" (UID: \"f8eba014-2735-4189-a982-e33413ce8e4b\") "
Oct 30 22:40:15.014041 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:15.013841    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8eba014-2735-4189-a982-e33413ce8e4b-kube-api-access-2qt4l\" (UniqueName: \"kubernetes.io/projected/f8eba014-2735-4189-a982-e33413ce8e4b-kube-api-access-2qt4l\") pod \"f8eba014-2735-4189-a982-e33413ce8e4b\" (UID: \"f8eba014-2735-4189-a982-e33413ce8e4b\") "
Oct 30 22:40:15.014041 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:15.013865    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f8eba014-2735-4189-a982-e33413ce8e4b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8eba014-2735-4189-a982-e33413ce8e4b-istio-podinfo\") pod \"f8eba014-2735-4189-a982-e33413ce8e4b\" (UID: \"f8eba014-2735-4189-a982-e33413ce8e4b\") "
Oct 30 22:40:15.014041 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:15.013942    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8eba014-2735-4189-a982-e33413ce8e4b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8eba014-2735-4189-a982-e33413ce8e4b-istio-data\") pod \"f8eba014-2735-4189-a982-e33413ce8e4b\" (UID: \"f8eba014-2735-4189-a982-e33413ce8e4b\") "
Oct 30 22:40:15.014041 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:15.013969    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8eba014-2735-4189-a982-e33413ce8e4b-credential-socket" pod "f8eba014-2735-4189-a982-e33413ce8e4b" (UID: "f8eba014-2735-4189-a982-e33413ce8e4b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:40:15.014312 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:15.014050    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8eba014-2735-4189-a982-e33413ce8e4b-workload-socket" pod "f8eba014-2735-4189-a982-e33413ce8e4b" (UID: "f8eba014-2735-4189-a982-e33413ce8e4b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:40:15.014312 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:15.014057    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8eba014-2735-4189-a982-e33413ce8e4b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8eba014-2735-4189-a982-e33413ce8e4b-istiod-ca-cert\") pod \"f8eba014-2735-4189-a982-e33413ce8e4b\" (UID: \"f8eba014-2735-4189-a982-e33413ce8e4b\") "
Oct 30 22:40:15.014312 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:15.014167    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8eba014-2735-4189-a982-e33413ce8e4b-workload-certs" pod "f8eba014-2735-4189-a982-e33413ce8e4b" (UID: "f8eba014-2735-4189-a982-e33413ce8e4b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:40:15.014312 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:15.014204    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8eba014-2735-4189-a982-e33413ce8e4b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8eba014-2735-4189-a982-e33413ce8e4b-istio-envoy\") pod \"f8eba014-2735-4189-a982-e33413ce8e4b\" (UID: \"f8eba014-2735-4189-a982-e33413ce8e4b\") "
Oct 30 22:40:15.014312 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:15.014293    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8eba014-2735-4189-a982-e33413ce8e4b-istio-data" pod "f8eba014-2735-4189-a982-e33413ce8e4b" (UID: "f8eba014-2735-4189-a982-e33413ce8e4b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:40:15.014576 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:15.014434    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8eba014-2735-4189-a982-e33413ce8e4b-istiod-ca-cert" pod "f8eba014-2735-4189-a982-e33413ce8e4b" (UID: "f8eba014-2735-4189-a982-e33413ce8e4b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:40:15.014640 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:15.014589    5678 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8eba014-2735-4189-a982-e33413ce8e4b-workload-certs\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:15.014640 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:15.014612    5678 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8eba014-2735-4189-a982-e33413ce8e4b-istio-data\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:15.014640 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:15.014626    5678 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8eba014-2735-4189-a982-e33413ce8e4b-istiod-ca-cert\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:15.014761 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:15.014641    5678 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8eba014-2735-4189-a982-e33413ce8e4b-credential-socket\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:15.014761 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:15.014656    5678 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8eba014-2735-4189-a982-e33413ce8e4b-workload-socket\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:15.016216 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:15.016173    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8eba014-2735-4189-a982-e33413ce8e4b-istio-envoy" pod "f8eba014-2735-4189-a982-e33413ce8e4b" (UID: "f8eba014-2735-4189-a982-e33413ce8e4b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:40:15.016216 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:15.016195    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f8eba014-2735-4189-a982-e33413ce8e4b-istio-podinfo" pod "f8eba014-2735-4189-a982-e33413ce8e4b" (UID: "f8eba014-2735-4189-a982-e33413ce8e4b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:40:15.016502 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:15.016482    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8eba014-2735-4189-a982-e33413ce8e4b-kube-api-access-2qt4l" pod "f8eba014-2735-4189-a982-e33413ce8e4b" (UID: "f8eba014-2735-4189-a982-e33413ce8e4b"). InnerVolumeSpecName "kube-api-access-2qt4l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:40:15.016553 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:15.016501    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8eba014-2735-4189-a982-e33413ce8e4b-istio-token" pod "f8eba014-2735-4189-a982-e33413ce8e4b" (UID: "f8eba014-2735-4189-a982-e33413ce8e4b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:40:15.116172 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:15.116142    5678 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8eba014-2735-4189-a982-e33413ce8e4b-istio-envoy\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:15.116172 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:15.116168    5678 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8eba014-2735-4189-a982-e33413ce8e4b-istio-token\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:15.116172 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:15.116179    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qt4l\" (UniqueName: \"kubernetes.io/projected/f8eba014-2735-4189-a982-e33413ce8e4b-kube-api-access-2qt4l\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:15.116390 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:15.116188    5678 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8eba014-2735-4189-a982-e33413ce8e4b-istio-podinfo\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:15.124946 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:15.124913    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-fb0b93/distinct-2-w6-fb0b93-kserve-7d65bb4698-m6tkh"]
Oct 30 22:40:15.126959 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:15.126932    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-fb0b93/simulator-w6-fb0b93-kserve-55b784b479-588bn"]
Oct 30 22:40:15.610707 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:15.610658    5678 generic.go:356] "Generic (PLEG): container finished" podID="f8eba014-2735-4189-a982-e33413ce8e4b" containerID="38afa7c023ddcdbab4c782283d3887435ad57de709f00e5888f0dcd9fd7e9995" exitCode=0
Oct 30 22:40:15.611187 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:15.610711    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-07e9c8-openshift-default-5944f97d49-m76c9" event={"ID":"f8eba014-2735-4189-a982-e33413ce8e4b","Type":"ContainerDied","Data":"38afa7c023ddcdbab4c782283d3887435ad57de709f00e5888f0dcd9fd7e9995"}
Oct 30 22:40:15.611187 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:15.610749    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-07e9c8-openshift-default-5944f97d49-m76c9"
Oct 30 22:40:15.611187 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:15.610795    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-07e9c8-openshift-default-5944f97d49-m76c9" event={"ID":"f8eba014-2735-4189-a982-e33413ce8e4b","Type":"ContainerDied","Data":"283b845b3778e9357d462d2cecf8e80b7817bff321185ed875cdd807b151ca66"}
Oct 30 22:40:15.611187 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:15.610827    5678 scope.go:122] "RemoveContainer" containerID="38afa7c023ddcdbab4c782283d3887435ad57de709f00e5888f0dcd9fd7e9995"
Oct 30 22:40:15.622132 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:15.622108    5678 scope.go:122] "RemoveContainer" containerID="a5e150eb8bd120fca6311562b9ea5e1f23c79a3670a3aa5f575f15df9856d9a1"
Oct 30 22:40:15.638896 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:15.638865    5678 scope.go:122] "RemoveContainer" containerID="38afa7c023ddcdbab4c782283d3887435ad57de709f00e5888f0dcd9fd7e9995"
Oct 30 22:40:15.639232 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:40:15.639202    5678 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38afa7c023ddcdbab4c782283d3887435ad57de709f00e5888f0dcd9fd7e9995\": container with ID starting with 38afa7c023ddcdbab4c782283d3887435ad57de709f00e5888f0dcd9fd7e9995 not found: ID does not exist" containerID="38afa7c023ddcdbab4c782283d3887435ad57de709f00e5888f0dcd9fd7e9995"
Oct 30 22:40:15.639302 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:15.639251    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38afa7c023ddcdbab4c782283d3887435ad57de709f00e5888f0dcd9fd7e9995"} err="failed to get container status \"38afa7c023ddcdbab4c782283d3887435ad57de709f00e5888f0dcd9fd7e9995\": rpc error: code = NotFound desc = could not find container \"38afa7c023ddcdbab4c782283d3887435ad57de709f00e5888f0dcd9fd7e9995\": container with ID starting with 38afa7c023ddcdbab4c782283d3887435ad57de709f00e5888f0dcd9fd7e9995 not found: ID does not exist"
Oct 30 22:40:15.639302 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:15.639283    5678 scope.go:122] "RemoveContainer" containerID="a5e150eb8bd120fca6311562b9ea5e1f23c79a3670a3aa5f575f15df9856d9a1"
Oct 30 22:40:15.639711 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:40:15.639675    5678 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5e150eb8bd120fca6311562b9ea5e1f23c79a3670a3aa5f575f15df9856d9a1\": container with ID starting with a5e150eb8bd120fca6311562b9ea5e1f23c79a3670a3aa5f575f15df9856d9a1 not found: ID does not exist" containerID="a5e150eb8bd120fca6311562b9ea5e1f23c79a3670a3aa5f575f15df9856d9a1"
Oct 30 22:40:15.639776 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:15.639731    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5e150eb8bd120fca6311562b9ea5e1f23c79a3670a3aa5f575f15df9856d9a1"} err="failed to get container status \"a5e150eb8bd120fca6311562b9ea5e1f23c79a3670a3aa5f575f15df9856d9a1\": rpc error: code = NotFound desc = could not find container \"a5e150eb8bd120fca6311562b9ea5e1f23c79a3670a3aa5f575f15df9856d9a1\": container with ID starting with a5e150eb8bd120fca6311562b9ea5e1f23c79a3670a3aa5f575f15df9856d9a1 not found: ID does not exist"
Oct 30 22:40:15.768508 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:15.768475    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-07e9c8-openshift-default-5944f97d49-m76c9"]
Oct 30 22:40:15.773165 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:15.773137    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-07e9c8-openshift-default-5944f97d49-m76c9"]
Oct 30 22:40:16.259701 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:16.259673    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-943d8f-5bf76c64cb-2rgrl"]
Oct 30 22:40:16.259936 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:16.259900    5678 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-943d8f-5bf76c64cb-2rgrl" podUID="a9fd32b5-0235-4b94-af2d-7c9bfbe2df09" containerName="maas-api" containerID="cri-o://f9ab639922e7443a9778c19f2f562146730975743027b90fce6a4cb46c37ae72" gracePeriod=30
Oct 30 22:40:16.511852 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:16.511797    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-943d8f-5bf76c64cb-2rgrl"
Oct 30 22:40:16.522135 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:16.522109    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8eba014-2735-4189-a982-e33413ce8e4b" path="/var/lib/kubelet/pods/f8eba014-2735-4189-a982-e33413ce8e4b/volumes"
Oct 30 22:40:16.619708 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:16.619673    5678 generic.go:356] "Generic (PLEG): container finished" podID="a9fd32b5-0235-4b94-af2d-7c9bfbe2df09" containerID="f9ab639922e7443a9778c19f2f562146730975743027b90fce6a4cb46c37ae72" exitCode=0
Oct 30 22:40:16.620108 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:16.619717    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-943d8f-5bf76c64cb-2rgrl"
Oct 30 22:40:16.620108 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:16.619757    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-943d8f-5bf76c64cb-2rgrl" event={"ID":"a9fd32b5-0235-4b94-af2d-7c9bfbe2df09","Type":"ContainerDied","Data":"f9ab639922e7443a9778c19f2f562146730975743027b90fce6a4cb46c37ae72"}
Oct 30 22:40:16.620108 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:16.619808    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-943d8f-5bf76c64cb-2rgrl" event={"ID":"a9fd32b5-0235-4b94-af2d-7c9bfbe2df09","Type":"ContainerDied","Data":"856ff1fe5fceac15974101d1066f24479d74f7661d9850a2b0a7bcd595a9d229"}
Oct 30 22:40:16.620108 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:16.619832    5678 scope.go:122] "RemoveContainer" containerID="f9ab639922e7443a9778c19f2f562146730975743027b90fce6a4cb46c37ae72"
Oct 30 22:40:16.630375 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:16.630346    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9fd32b5-0235-4b94-af2d-7c9bfbe2df09-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9fd32b5-0235-4b94-af2d-7c9bfbe2df09-maas-api-tls\") pod \"a9fd32b5-0235-4b94-af2d-7c9bfbe2df09\" (UID: \"a9fd32b5-0235-4b94-af2d-7c9bfbe2df09\") "
Oct 30 22:40:16.630535 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:16.630436    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9fd32b5-0235-4b94-af2d-7c9bfbe2df09-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9fd32b5-0235-4b94-af2d-7c9bfbe2df09-openshift-service-ca\") pod \"a9fd32b5-0235-4b94-af2d-7c9bfbe2df09\" (UID: \"a9fd32b5-0235-4b94-af2d-7c9bfbe2df09\") "
Oct 30 22:40:16.630593 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:16.630545    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9fd32b5-0235-4b94-af2d-7c9bfbe2df09-kube-api-access-2ddks\" (UniqueName: \"kubernetes.io/projected/a9fd32b5-0235-4b94-af2d-7c9bfbe2df09-kube-api-access-2ddks\") pod \"a9fd32b5-0235-4b94-af2d-7c9bfbe2df09\" (UID: \"a9fd32b5-0235-4b94-af2d-7c9bfbe2df09\") "
Oct 30 22:40:16.630636 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:16.630598    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9fd32b5-0235-4b94-af2d-7c9bfbe2df09-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9fd32b5-0235-4b94-af2d-7c9bfbe2df09-metrics-certs\") pod \"a9fd32b5-0235-4b94-af2d-7c9bfbe2df09\" (UID: \"a9fd32b5-0235-4b94-af2d-7c9bfbe2df09\") "
Oct 30 22:40:16.630850 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:16.630820    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9fd32b5-0235-4b94-af2d-7c9bfbe2df09-openshift-service-ca" pod "a9fd32b5-0235-4b94-af2d-7c9bfbe2df09" (UID: "a9fd32b5-0235-4b94-af2d-7c9bfbe2df09"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:40:16.631033 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:16.631010    5678 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9fd32b5-0235-4b94-af2d-7c9bfbe2df09-openshift-service-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:16.631181 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:16.631115    5678 scope.go:122] "RemoveContainer" containerID="f9ab639922e7443a9778c19f2f562146730975743027b90fce6a4cb46c37ae72"
Oct 30 22:40:16.631520 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:40:16.631494    5678 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9ab639922e7443a9778c19f2f562146730975743027b90fce6a4cb46c37ae72\": container with ID starting with f9ab639922e7443a9778c19f2f562146730975743027b90fce6a4cb46c37ae72 not found: ID does not exist" containerID="f9ab639922e7443a9778c19f2f562146730975743027b90fce6a4cb46c37ae72"
Oct 30 22:40:16.631637 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:16.631532    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9ab639922e7443a9778c19f2f562146730975743027b90fce6a4cb46c37ae72"} err="failed to get container status \"f9ab639922e7443a9778c19f2f562146730975743027b90fce6a4cb46c37ae72\": rpc error: code = NotFound desc = could not find container \"f9ab639922e7443a9778c19f2f562146730975743027b90fce6a4cb46c37ae72\": container with ID starting with f9ab639922e7443a9778c19f2f562146730975743027b90fce6a4cb46c37ae72 not found: ID does not exist"
Oct 30 22:40:16.633127 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:16.633088    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9fd32b5-0235-4b94-af2d-7c9bfbe2df09-maas-api-tls" pod "a9fd32b5-0235-4b94-af2d-7c9bfbe2df09" (UID: "a9fd32b5-0235-4b94-af2d-7c9bfbe2df09"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:40:16.633248 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:16.633206    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9fd32b5-0235-4b94-af2d-7c9bfbe2df09-metrics-certs" pod "a9fd32b5-0235-4b94-af2d-7c9bfbe2df09" (UID: "a9fd32b5-0235-4b94-af2d-7c9bfbe2df09"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:40:16.633507 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:16.633481    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9fd32b5-0235-4b94-af2d-7c9bfbe2df09-kube-api-access-2ddks" pod "a9fd32b5-0235-4b94-af2d-7c9bfbe2df09" (UID: "a9fd32b5-0235-4b94-af2d-7c9bfbe2df09"). InnerVolumeSpecName "kube-api-access-2ddks". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:40:16.731756 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:16.731722    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9fd32b5-0235-4b94-af2d-7c9bfbe2df09-maas-api-tls\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:16.731756 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:16.731752    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ddks\" (UniqueName: \"kubernetes.io/projected/a9fd32b5-0235-4b94-af2d-7c9bfbe2df09-kube-api-access-2ddks\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:16.731931 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:16.731769    5678 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9fd32b5-0235-4b94-af2d-7c9bfbe2df09-metrics-certs\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:17.039727 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:17.039695    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-943d8f-5bf76c64cb-2rgrl"]
Oct 30 22:40:17.086757 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:17.086728    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-943d8f-5bf76c64cb-2rgrl"]
Oct 30 22:40:18.523697 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:18.523665    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9fd32b5-0235-4b94-af2d-7c9bfbe2df09" path="/var/lib/kubelet/pods/a9fd32b5-0235-4b94-af2d-7c9bfbe2df09/volumes"
Oct 30 22:40:19.074942 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:19.074919    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-07e9c8-5bd8cdfbff-h8vtx"
Oct 30 22:40:19.157696 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:19.157672    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab8b95b5-8eca-405d-bfa9-195e5b4f68f7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab8b95b5-8eca-405d-bfa9-195e5b4f68f7-plugins-config-volume\") pod \"ab8b95b5-8eca-405d-bfa9-195e5b4f68f7\" (UID: \"ab8b95b5-8eca-405d-bfa9-195e5b4f68f7\") "
Oct 30 22:40:19.157837 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:19.157706    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab8b95b5-8eca-405d-bfa9-195e5b4f68f7-kube-api-access-fcwh4\" (UniqueName: \"kubernetes.io/projected/ab8b95b5-8eca-405d-bfa9-195e5b4f68f7-kube-api-access-fcwh4\") pod \"ab8b95b5-8eca-405d-bfa9-195e5b4f68f7\" (UID: \"ab8b95b5-8eca-405d-bfa9-195e5b4f68f7\") "
Oct 30 22:40:19.158042 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:19.158021    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab8b95b5-8eca-405d-bfa9-195e5b4f68f7-plugins-config-volume" pod "ab8b95b5-8eca-405d-bfa9-195e5b4f68f7" (UID: "ab8b95b5-8eca-405d-bfa9-195e5b4f68f7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:40:19.159693 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:19.159670    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab8b95b5-8eca-405d-bfa9-195e5b4f68f7-kube-api-access-fcwh4" pod "ab8b95b5-8eca-405d-bfa9-195e5b4f68f7" (UID: "ab8b95b5-8eca-405d-bfa9-195e5b4f68f7"). InnerVolumeSpecName "kube-api-access-fcwh4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:40:19.170768 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:19.170740    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-07e9c8-5f45d5ff75-z224x"
Oct 30 22:40:19.258972 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:19.258948    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47873f24-8e29-439b-8017-776ece323a1c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47873f24-8e29-439b-8017-776ece323a1c-plugins-config-volume\") pod \"47873f24-8e29-439b-8017-776ece323a1c\" (UID: \"47873f24-8e29-439b-8017-776ece323a1c\") "
Oct 30 22:40:19.259150 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:19.259039    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47873f24-8e29-439b-8017-776ece323a1c-kube-api-access-rlrch\" (UniqueName: \"kubernetes.io/projected/47873f24-8e29-439b-8017-776ece323a1c-kube-api-access-rlrch\") pod \"47873f24-8e29-439b-8017-776ece323a1c\" (UID: \"47873f24-8e29-439b-8017-776ece323a1c\") "
Oct 30 22:40:19.259277 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:19.259263    5678 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab8b95b5-8eca-405d-bfa9-195e5b4f68f7-plugins-config-volume\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:19.259355 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:19.259277    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fcwh4\" (UniqueName: \"kubernetes.io/projected/ab8b95b5-8eca-405d-bfa9-195e5b4f68f7-kube-api-access-fcwh4\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:19.259435 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:19.259340    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47873f24-8e29-439b-8017-776ece323a1c-plugins-config-volume" pod "47873f24-8e29-439b-8017-776ece323a1c" (UID: "47873f24-8e29-439b-8017-776ece323a1c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:40:19.261074 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:19.261047    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47873f24-8e29-439b-8017-776ece323a1c-kube-api-access-rlrch" pod "47873f24-8e29-439b-8017-776ece323a1c" (UID: "47873f24-8e29-439b-8017-776ece323a1c"). InnerVolumeSpecName "kube-api-access-rlrch". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:40:19.360855 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:19.360781    5678 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47873f24-8e29-439b-8017-776ece323a1c-plugins-config-volume\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:19.360855 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:19.360808    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rlrch\" (UniqueName: \"kubernetes.io/projected/47873f24-8e29-439b-8017-776ece323a1c-kube-api-access-rlrch\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:19.641165 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:19.641132    5678 generic.go:356] "Generic (PLEG): container finished" podID="ab8b95b5-8eca-405d-bfa9-195e5b4f68f7" containerID="c23c11290c0bf58481911bdb94015c88696afc359ce77e5cdd6fa848dc598a3c" exitCode=0
Oct 30 22:40:19.641636 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:19.641237    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-07e9c8-5bd8cdfbff-h8vtx"
Oct 30 22:40:19.641636 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:19.641238    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-07e9c8-5bd8cdfbff-h8vtx" event={"ID":"ab8b95b5-8eca-405d-bfa9-195e5b4f68f7","Type":"ContainerDied","Data":"c23c11290c0bf58481911bdb94015c88696afc359ce77e5cdd6fa848dc598a3c"}
Oct 30 22:40:19.641636 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:19.641382    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-07e9c8-5bd8cdfbff-h8vtx" event={"ID":"ab8b95b5-8eca-405d-bfa9-195e5b4f68f7","Type":"ContainerDied","Data":"39309ac761d1d7d22dc6d5ddfbb2cce1a8dfc9880b89d74e20c69957fe920fff"}
Oct 30 22:40:19.641636 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:19.641434    5678 scope.go:122] "RemoveContainer" containerID="c23c11290c0bf58481911bdb94015c88696afc359ce77e5cdd6fa848dc598a3c"
Oct 30 22:40:19.642747 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:19.642718    5678 generic.go:356] "Generic (PLEG): container finished" podID="47873f24-8e29-439b-8017-776ece323a1c" containerID="4fd044d0bf9c012d99f4d112c479b74f487d514963982abe3a65523687363140" exitCode=0
Oct 30 22:40:19.642888 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:19.642745    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-07e9c8-5f45d5ff75-z224x" event={"ID":"47873f24-8e29-439b-8017-776ece323a1c","Type":"ContainerDied","Data":"4fd044d0bf9c012d99f4d112c479b74f487d514963982abe3a65523687363140"}
Oct 30 22:40:19.642888 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:19.642762    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-07e9c8-5f45d5ff75-z224x"
Oct 30 22:40:19.642888 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:19.642787    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-07e9c8-5f45d5ff75-z224x" event={"ID":"47873f24-8e29-439b-8017-776ece323a1c","Type":"ContainerDied","Data":"7aa02f39ac9bfdff442305f301ff58ae88afd855ddeb2f38b9bf40e628583691"}
Oct 30 22:40:19.651674 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:19.651644    5678 scope.go:122] "RemoveContainer" containerID="c23c11290c0bf58481911bdb94015c88696afc359ce77e5cdd6fa848dc598a3c"
Oct 30 22:40:19.651989 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:40:19.651960    5678 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c23c11290c0bf58481911bdb94015c88696afc359ce77e5cdd6fa848dc598a3c\": container with ID starting with c23c11290c0bf58481911bdb94015c88696afc359ce77e5cdd6fa848dc598a3c not found: ID does not exist" containerID="c23c11290c0bf58481911bdb94015c88696afc359ce77e5cdd6fa848dc598a3c"
Oct 30 22:40:19.652086 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:19.651999    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c23c11290c0bf58481911bdb94015c88696afc359ce77e5cdd6fa848dc598a3c"} err="failed to get container status \"c23c11290c0bf58481911bdb94015c88696afc359ce77e5cdd6fa848dc598a3c\": rpc error: code = NotFound desc = could not find container \"c23c11290c0bf58481911bdb94015c88696afc359ce77e5cdd6fa848dc598a3c\": container with ID starting with c23c11290c0bf58481911bdb94015c88696afc359ce77e5cdd6fa848dc598a3c not found: ID does not exist"
Oct 30 22:40:19.652086 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:19.652019    5678 scope.go:122] "RemoveContainer" containerID="4fd044d0bf9c012d99f4d112c479b74f487d514963982abe3a65523687363140"
Oct 30 22:40:19.659894 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:19.659879    5678 scope.go:122] "RemoveContainer" containerID="4fd044d0bf9c012d99f4d112c479b74f487d514963982abe3a65523687363140"
Oct 30 22:40:19.660141 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:40:19.660124    5678 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fd044d0bf9c012d99f4d112c479b74f487d514963982abe3a65523687363140\": container with ID starting with 4fd044d0bf9c012d99f4d112c479b74f487d514963982abe3a65523687363140 not found: ID does not exist" containerID="4fd044d0bf9c012d99f4d112c479b74f487d514963982abe3a65523687363140"
Oct 30 22:40:19.660192 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:19.660149    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fd044d0bf9c012d99f4d112c479b74f487d514963982abe3a65523687363140"} err="failed to get container status \"4fd044d0bf9c012d99f4d112c479b74f487d514963982abe3a65523687363140\": rpc error: code = NotFound desc = could not find container \"4fd044d0bf9c012d99f4d112c479b74f487d514963982abe3a65523687363140\": container with ID starting with 4fd044d0bf9c012d99f4d112c479b74f487d514963982abe3a65523687363140 not found: ID does not exist"
Oct 30 22:40:19.802119 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:19.802090    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-07e9c8-5f45d5ff75-z224x"]
Oct 30 22:40:19.849018 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:19.848991    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-07e9c8-5f45d5ff75-z224x"]
Oct 30 22:40:19.912480 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:19.912418    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-07e9c8-5bd8cdfbff-h8vtx"]
Oct 30 22:40:19.917097 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:19.917070    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-07e9c8-5bd8cdfbff-h8vtx"]
Oct 30 22:40:20.522655 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:20.522620    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47873f24-8e29-439b-8017-776ece323a1c" path="/var/lib/kubelet/pods/47873f24-8e29-439b-8017-776ece323a1c/volumes"
Oct 30 22:40:20.523035 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:20.523021    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab8b95b5-8eca-405d-bfa9-195e5b4f68f7" path="/var/lib/kubelet/pods/ab8b95b5-8eca-405d-bfa9-195e5b4f68f7/volumes"
Oct 30 22:40:21.138714 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:21.138678    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-fb0b93-7fb666d8f9-wr46f"]
Oct 30 22:40:26.139495 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:26.139417    5678 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-fb0b93-7fb666d8f9-wr46f" podUID="24066223-6a66-43ef-a85d-83183bc37fa7" containerName="payload-pre-processing" containerID="cri-o://caef6a84c7871725c67e240f5e7d0a2b96ce07698edb2d00fd3566e1e8e78a98" gracePeriod=25
Oct 30 22:40:28.227313 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:28.227279    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a00d9b-hj4wb"]
Oct 30 22:40:28.227712 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:28.227669    5678 cpu_manager.go:414] "containerMap: removing container" podUID="09ddef89-ac05-4e76-b7ef-449236df8900" containerName="maas-api"
Oct 30 22:40:28.227712 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:28.227681    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09ddef89-ac05-4e76-b7ef-449236df8900" containerName="maas-api"
Oct 30 22:40:28.227712 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:28.227696    5678 cpu_manager.go:414] "containerMap: removing container" podUID="a9fd32b5-0235-4b94-af2d-7c9bfbe2df09" containerName="maas-api"
Oct 30 22:40:28.227712 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:28.227701    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9fd32b5-0235-4b94-af2d-7c9bfbe2df09" containerName="maas-api"
Oct 30 22:40:28.227712 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:28.227713    5678 cpu_manager.go:414] "containerMap: removing container" podUID="95ef4304-6677-4325-8396-31575317404c" containerName="payload-processing"
Oct 30 22:40:28.227887 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:28.227718    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95ef4304-6677-4325-8396-31575317404c" containerName="payload-processing"
Oct 30 22:40:28.227887 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:28.227726    5678 cpu_manager.go:414] "containerMap: removing container" podUID="5d40decd-0eb0-4636-a8fe-06bbafd0484e" containerName="istio-proxy"
Oct 30 22:40:28.227887 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:28.227731    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d40decd-0eb0-4636-a8fe-06bbafd0484e" containerName="istio-proxy"
Oct 30 22:40:28.227887 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:28.227741    5678 cpu_manager.go:414] "containerMap: removing container" podUID="b1db85b2-f8d2-42b9-8731-4e9e9bff3f38" containerName="payload-processing"
Oct 30 22:40:28.227887 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:28.227746    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1db85b2-f8d2-42b9-8731-4e9e9bff3f38" containerName="payload-processing"
Oct 30 22:40:28.227887 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:28.227753    5678 cpu_manager.go:414] "containerMap: removing container" podUID="e325caa7-d4fb-4a0e-9fe0-ee95742278b3" containerName="istio-proxy"
Oct 30 22:40:28.227887 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:28.227758    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e325caa7-d4fb-4a0e-9fe0-ee95742278b3" containerName="istio-proxy"
Oct 30 22:40:28.227887 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:28.227765    5678 cpu_manager.go:414] "containerMap: removing container" podUID="99f99481-73d0-4fb4-9c88-12ee143a6904" containerName="istio-proxy"
Oct 30 22:40:28.227887 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:28.227769    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99f99481-73d0-4fb4-9c88-12ee143a6904" containerName="istio-proxy"
Oct 30 22:40:28.227887 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:28.227774    5678 cpu_manager.go:414] "containerMap: removing container" podUID="f1691924-5354-4690-8709-6a20bfd81a7e" containerName="istio-proxy"
Oct 30 22:40:28.227887 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:28.227778    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1691924-5354-4690-8709-6a20bfd81a7e" containerName="istio-proxy"
Oct 30 22:40:28.227887 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:28.227787    5678 cpu_manager.go:414] "containerMap: removing container" podUID="01b7d4f9-77ed-4292-a572-3a910bc8521e" containerName="payload-pre-processing"
Oct 30 22:40:28.227887 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:28.227791    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01b7d4f9-77ed-4292-a572-3a910bc8521e" containerName="payload-pre-processing"
Oct 30 22:40:28.227887 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:28.227798    5678 cpu_manager.go:414] "containerMap: removing container" podUID="ab8b95b5-8eca-405d-bfa9-195e5b4f68f7" containerName="payload-processing"
Oct 30 22:40:28.227887 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:28.227803    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab8b95b5-8eca-405d-bfa9-195e5b4f68f7" containerName="payload-processing"
Oct 30 22:40:28.227887 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:28.227811    5678 cpu_manager.go:414] "containerMap: removing container" podUID="699c4653-4da2-4b6f-965e-d34b632a8340" containerName="maas-api"
Oct 30 22:40:28.227887 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:28.227816    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="699c4653-4da2-4b6f-965e-d34b632a8340" containerName="maas-api"
Oct 30 22:40:28.227887 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:28.227824    5678 cpu_manager.go:414] "containerMap: removing container" podUID="c8157ccd-c3b7-4b4b-8dcb-9a8881eaff26" containerName="main"
Oct 30 22:40:28.227887 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:28.227829    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8157ccd-c3b7-4b4b-8dcb-9a8881eaff26" containerName="main"
Oct 30 22:40:28.227887 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:28.227837    5678 cpu_manager.go:414] "containerMap: removing container" podUID="f8eba014-2735-4189-a982-e33413ce8e4b" containerName="istio-proxy"
Oct 30 22:40:28.227887 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:28.227843    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8eba014-2735-4189-a982-e33413ce8e4b" containerName="istio-proxy"
Oct 30 22:40:28.227887 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:28.227849    5678 cpu_manager.go:414] "containerMap: removing container" podUID="64a0ba20-2130-4df1-b147-28a52a7d0e62" containerName="istio-proxy"
Oct 30 22:40:28.227887 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:28.227853    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64a0ba20-2130-4df1-b147-28a52a7d0e62" containerName="istio-proxy"
Oct 30 22:40:28.227887 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:28.227859    5678 cpu_manager.go:414] "containerMap: removing container" podUID="64a0ba20-2130-4df1-b147-28a52a7d0e62" containerName="istio-proxy"
Oct 30 22:40:28.227887 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:28.227864    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64a0ba20-2130-4df1-b147-28a52a7d0e62" containerName="istio-proxy"
Oct 30 22:40:28.227887 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:28.227872    5678 cpu_manager.go:414] "containerMap: removing container" podUID="bdeffce0-5736-4ac8-89b1-cfdd262fd5c7" containerName="payload-processing"
Oct 30 22:40:28.227887 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:28.227876    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bdeffce0-5736-4ac8-89b1-cfdd262fd5c7" containerName="payload-processing"
Oct 30 22:40:28.227887 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:28.227885    5678 cpu_manager.go:414] "containerMap: removing container" podUID="40138fb9-9a75-4458-a351-11eb61c4a1e3" containerName="main"
Oct 30 22:40:28.227887 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:28.227890    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40138fb9-9a75-4458-a351-11eb61c4a1e3" containerName="main"
Oct 30 22:40:28.227887 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:28.227896    5678 cpu_manager.go:414] "containerMap: removing container" podUID="f1e95a6d-29c6-4c95-b537-1037089dcab8" containerName="payload-pre-processing"
Oct 30 22:40:28.227887 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:28.227901    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1e95a6d-29c6-4c95-b537-1037089dcab8" containerName="payload-pre-processing"
Oct 30 22:40:28.228808 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:28.227907    5678 cpu_manager.go:414] "containerMap: removing container" podUID="47873f24-8e29-439b-8017-776ece323a1c" containerName="payload-pre-processing"
Oct 30 22:40:28.228808 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:28.227912    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47873f24-8e29-439b-8017-776ece323a1c" containerName="payload-pre-processing"
Oct 30 22:40:28.228808 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:28.227920    5678 cpu_manager.go:414] "containerMap: removing container" podUID="0e40fbc5-668a-4917-9344-dea69c625d26" containerName="maas-api"
Oct 30 22:40:28.228808 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:28.227925    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e40fbc5-668a-4917-9344-dea69c625d26" containerName="maas-api"
Oct 30 22:40:28.228808 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:28.227933    5678 cpu_manager.go:414] "containerMap: removing container" podUID="f1691924-5354-4690-8709-6a20bfd81a7e" containerName="istio-proxy"
Oct 30 22:40:28.228808 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:28.227938    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1691924-5354-4690-8709-6a20bfd81a7e" containerName="istio-proxy"
Oct 30 22:40:28.228808 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:28.227944    5678 cpu_manager.go:414] "containerMap: removing container" podUID="c01c1a4e-7d4d-45ca-b92e-c1cdd9013059" containerName="maas-api"
Oct 30 22:40:28.228808 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:28.227948    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c01c1a4e-7d4d-45ca-b92e-c1cdd9013059" containerName="maas-api"
Oct 30 22:40:28.228808 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:28.228008    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="01b7d4f9-77ed-4292-a572-3a910bc8521e" containerName="payload-pre-processing"
Oct 30 22:40:28.228808 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:28.228017    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab8b95b5-8eca-405d-bfa9-195e5b4f68f7" containerName="payload-processing"
Oct 30 22:40:28.228808 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:28.228023    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="699c4653-4da2-4b6f-965e-d34b632a8340" containerName="maas-api"
Oct 30 22:40:28.228808 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:28.228029    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8157ccd-c3b7-4b4b-8dcb-9a8881eaff26" containerName="main"
Oct 30 22:40:28.228808 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:28.228034    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="99f99481-73d0-4fb4-9c88-12ee143a6904" containerName="istio-proxy"
Oct 30 22:40:28.228808 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:28.228038    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8eba014-2735-4189-a982-e33413ce8e4b" containerName="istio-proxy"
Oct 30 22:40:28.228808 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:28.228043    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="64a0ba20-2130-4df1-b147-28a52a7d0e62" containerName="istio-proxy"
Oct 30 22:40:28.228808 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:28.228049    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="64a0ba20-2130-4df1-b147-28a52a7d0e62" containerName="istio-proxy"
Oct 30 22:40:28.228808 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:28.228055    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="bdeffce0-5736-4ac8-89b1-cfdd262fd5c7" containerName="payload-processing"
Oct 30 22:40:28.228808 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:28.228061    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="40138fb9-9a75-4458-a351-11eb61c4a1e3" containerName="main"
Oct 30 22:40:28.228808 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:28.228065    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1e95a6d-29c6-4c95-b537-1037089dcab8" containerName="payload-pre-processing"
Oct 30 22:40:28.228808 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:28.228070    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="47873f24-8e29-439b-8017-776ece323a1c" containerName="payload-pre-processing"
Oct 30 22:40:28.228808 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:28.228075    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8eba014-2735-4189-a982-e33413ce8e4b" containerName="istio-proxy"
Oct 30 22:40:28.228808 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:28.228080    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e40fbc5-668a-4917-9344-dea69c625d26" containerName="maas-api"
Oct 30 22:40:28.228808 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:28.228085    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1691924-5354-4690-8709-6a20bfd81a7e" containerName="istio-proxy"
Oct 30 22:40:28.228808 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:28.228089    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="c01c1a4e-7d4d-45ca-b92e-c1cdd9013059" containerName="maas-api"
Oct 30 22:40:28.228808 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:28.228095    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="e325caa7-d4fb-4a0e-9fe0-ee95742278b3" containerName="istio-proxy"
Oct 30 22:40:28.228808 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:28.228099    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="09ddef89-ac05-4e76-b7ef-449236df8900" containerName="maas-api"
Oct 30 22:40:28.228808 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:28.228104    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9fd32b5-0235-4b94-af2d-7c9bfbe2df09" containerName="maas-api"
Oct 30 22:40:28.228808 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:28.228109    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="95ef4304-6677-4325-8396-31575317404c" containerName="payload-processing"
Oct 30 22:40:28.228808 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:28.228114    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d40decd-0eb0-4636-a8fe-06bbafd0484e" containerName="istio-proxy"
Oct 30 22:40:28.228808 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:28.228119    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1db85b2-f8d2-42b9-8731-4e9e9bff3f38" containerName="payload-processing"
Oct 30 22:40:28.228808 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:28.228124    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1691924-5354-4690-8709-6a20bfd81a7e" containerName="istio-proxy"
Oct 30 22:40:28.232964 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:28.232942    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a00d9b-hj4wb"
Oct 30 22:40:28.287429 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:28.287387    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:40:28.287539 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:28.287455    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5rvcj\""
Oct 30 22:40:28.290547 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:28.290533    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:40:28.334618 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:28.334586    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:40:28.336714 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:28.336692    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a00d9b-hj4wb"]
Oct 30 22:40:28.342150 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:28.342128    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6ncp\" (UniqueName: \"kubernetes.io/projected/805b2e79-a950-415f-8f98-9c01061d0479-kube-api-access-c6ncp\") pod \"maas-api-revoke-keys-e2e-worker-w1-a00d9b-hj4wb\" (UID: \"805b2e79-a950-415f-8f98-9c01061d0479\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a00d9b-hj4wb"
Oct 30 22:40:28.342251 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:28.342158    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/805b2e79-a950-415f-8f98-9c01061d0479-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-a00d9b-hj4wb\" (UID: \"805b2e79-a950-415f-8f98-9c01061d0479\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a00d9b-hj4wb"
Oct 30 22:40:28.342305 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:28.342249    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/805b2e79-a950-415f-8f98-9c01061d0479-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-a00d9b-hj4wb\" (UID: \"805b2e79-a950-415f-8f98-9c01061d0479\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a00d9b-hj4wb"
Oct 30 22:40:28.443579 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:28.443552    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/805b2e79-a950-415f-8f98-9c01061d0479-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-a00d9b-hj4wb\" (UID: \"805b2e79-a950-415f-8f98-9c01061d0479\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a00d9b-hj4wb"
Oct 30 22:40:28.443737 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:28.443653    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6ncp\" (UniqueName: \"kubernetes.io/projected/805b2e79-a950-415f-8f98-9c01061d0479-kube-api-access-c6ncp\") pod \"maas-api-revoke-keys-e2e-worker-w1-a00d9b-hj4wb\" (UID: \"805b2e79-a950-415f-8f98-9c01061d0479\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a00d9b-hj4wb"
Oct 30 22:40:28.443737 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:28.443674    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/805b2e79-a950-415f-8f98-9c01061d0479-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-a00d9b-hj4wb\" (UID: \"805b2e79-a950-415f-8f98-9c01061d0479\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a00d9b-hj4wb"
Oct 30 22:40:28.444208 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:28.444185    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/805b2e79-a950-415f-8f98-9c01061d0479-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-a00d9b-hj4wb\" (UID: \"805b2e79-a950-415f-8f98-9c01061d0479\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a00d9b-hj4wb"
Oct 30 22:40:28.445872 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:28.445849    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/805b2e79-a950-415f-8f98-9c01061d0479-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-a00d9b-hj4wb\" (UID: \"805b2e79-a950-415f-8f98-9c01061d0479\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a00d9b-hj4wb"
Oct 30 22:40:28.502780 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:28.502711    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6ncp\" (UniqueName: \"kubernetes.io/projected/805b2e79-a950-415f-8f98-9c01061d0479-kube-api-access-c6ncp\") pod \"maas-api-revoke-keys-e2e-worker-w1-a00d9b-hj4wb\" (UID: \"805b2e79-a950-415f-8f98-9c01061d0479\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a00d9b-hj4wb"
Oct 30 22:40:28.545528 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:28.545497    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a00d9b-hj4wb"
Oct 30 22:40:28.926735 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:28.926704    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a00d9b-hj4wb"]
Oct 30 22:40:28.934570 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:40:28.934545    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod805b2e79_a950_415f_8f98_9c01061d0479.slice/crio-809f4ec1b802be1b86c5c8061f1ba6fab06848b5361afadd0b43d860429e3332 WatchSource:0}: Error finding container 809f4ec1b802be1b86c5c8061f1ba6fab06848b5361afadd0b43d860429e3332: Status 404 returned error can't find the container with id 809f4ec1b802be1b86c5c8061f1ba6fab06848b5361afadd0b43d860429e3332
Oct 30 22:40:29.705759 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:29.705720    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a00d9b-hj4wb" event={"ID":"805b2e79-a950-415f-8f98-9c01061d0479","Type":"ContainerStarted","Data":"60c78f33c65112c090858ae6419d3003b9b82d40344881d3a2700fb5570ddc59"}
Oct 30 22:40:29.706162 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:29.705818    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a00d9b-hj4wb" event={"ID":"805b2e79-a950-415f-8f98-9c01061d0479","Type":"ContainerStarted","Data":"809f4ec1b802be1b86c5c8061f1ba6fab06848b5361afadd0b43d860429e3332"}
Oct 30 22:40:29.816636 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:29.816588    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a00d9b-hj4wb" podStartSLOduration=1.81657652 podStartE2EDuration="1.81657652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:40:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:40:29.815666351 +0000 UTC m=+3007.419973904" watchObservedRunningTime="2026-10-30 22:40:29.81657652 +0000 UTC m=+3007.420884072"
Oct 30 22:40:30.146325 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:30.146267    5678 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-fb0b93/distinct-2-w6-fb0b93-kserve-7d65bb4698-m6tkh" podUID="6fd276ef-522b-4bb4-b101-0a9c3fe9ecd3" containerName="main" containerID="cri-o://8997f6d274894fb2ebc9077cd8197d93f8c588510a2364ea8dc174114d982ed5" gracePeriod=45
Oct 30 22:40:30.152427 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:30.152381    5678 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-fb0b93/simulator-w6-fb0b93-kserve-55b784b479-588bn" podUID="951a6f7e-a22e-40f2-ace2-1ce5d3bc9b49" containerName="main" containerID="cri-o://e5809108b6519b3887dd567430d21c8ad497ea330ec6fd117ba229c8435fbda5" gracePeriod=45
Oct 30 22:40:30.396784 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:30.396729    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-fb0b93/distinct-2-w6-fb0b93-kserve-7d65bb4698-m6tkh"
Oct 30 22:40:30.410947 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:30.410926    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-fb0b93/simulator-w6-fb0b93-kserve-55b784b479-588bn"
Oct 30 22:40:30.564497 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:30.564461    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/951a6f7e-a22e-40f2-ace2-1ce5d3bc9b49-dshm\" (UniqueName: \"kubernetes.io/empty-dir/951a6f7e-a22e-40f2-ace2-1ce5d3bc9b49-dshm\") pod \"951a6f7e-a22e-40f2-ace2-1ce5d3bc9b49\" (UID: \"951a6f7e-a22e-40f2-ace2-1ce5d3bc9b49\") "
Oct 30 22:40:30.564691 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:30.564525    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/951a6f7e-a22e-40f2-ace2-1ce5d3bc9b49-kube-api-access-jmpvb\" (UniqueName: \"kubernetes.io/projected/951a6f7e-a22e-40f2-ace2-1ce5d3bc9b49-kube-api-access-jmpvb\") pod \"951a6f7e-a22e-40f2-ace2-1ce5d3bc9b49\" (UID: \"951a6f7e-a22e-40f2-ace2-1ce5d3bc9b49\") "
Oct 30 22:40:30.564691 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:30.564557    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/951a6f7e-a22e-40f2-ace2-1ce5d3bc9b49-home\" (UniqueName: \"kubernetes.io/empty-dir/951a6f7e-a22e-40f2-ace2-1ce5d3bc9b49-home\") pod \"951a6f7e-a22e-40f2-ace2-1ce5d3bc9b49\" (UID: \"951a6f7e-a22e-40f2-ace2-1ce5d3bc9b49\") "
Oct 30 22:40:30.564691 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:30.564573    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6fd276ef-522b-4bb4-b101-0a9c3fe9ecd3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6fd276ef-522b-4bb4-b101-0a9c3fe9ecd3-tmp-dir\") pod \"6fd276ef-522b-4bb4-b101-0a9c3fe9ecd3\" (UID: \"6fd276ef-522b-4bb4-b101-0a9c3fe9ecd3\") "
Oct 30 22:40:30.564691 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:30.564617    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6fd276ef-522b-4bb4-b101-0a9c3fe9ecd3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6fd276ef-522b-4bb4-b101-0a9c3fe9ecd3-dshm\") pod \"6fd276ef-522b-4bb4-b101-0a9c3fe9ecd3\" (UID: \"6fd276ef-522b-4bb4-b101-0a9c3fe9ecd3\") "
Oct 30 22:40:30.564691 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:30.564655    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/951a6f7e-a22e-40f2-ace2-1ce5d3bc9b49-tls-certs\" (UniqueName: \"kubernetes.io/secret/951a6f7e-a22e-40f2-ace2-1ce5d3bc9b49-tls-certs\") pod \"951a6f7e-a22e-40f2-ace2-1ce5d3bc9b49\" (UID: \"951a6f7e-a22e-40f2-ace2-1ce5d3bc9b49\") "
Oct 30 22:40:30.565053 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:30.564697    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6fd276ef-522b-4bb4-b101-0a9c3fe9ecd3-tls-certs\" (UniqueName: \"kubernetes.io/secret/6fd276ef-522b-4bb4-b101-0a9c3fe9ecd3-tls-certs\") pod \"6fd276ef-522b-4bb4-b101-0a9c3fe9ecd3\" (UID: \"6fd276ef-522b-4bb4-b101-0a9c3fe9ecd3\") "
Oct 30 22:40:30.565053 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:30.564734    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6fd276ef-522b-4bb4-b101-0a9c3fe9ecd3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6fd276ef-522b-4bb4-b101-0a9c3fe9ecd3-model-cache\") pod \"6fd276ef-522b-4bb4-b101-0a9c3fe9ecd3\" (UID: \"6fd276ef-522b-4bb4-b101-0a9c3fe9ecd3\") "
Oct 30 22:40:30.565053 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:30.564765    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6fd276ef-522b-4bb4-b101-0a9c3fe9ecd3-home\" (UniqueName: \"kubernetes.io/empty-dir/6fd276ef-522b-4bb4-b101-0a9c3fe9ecd3-home\") pod \"6fd276ef-522b-4bb4-b101-0a9c3fe9ecd3\" (UID: \"6fd276ef-522b-4bb4-b101-0a9c3fe9ecd3\") "
Oct 30 22:40:30.565053 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:30.564850    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fd276ef-522b-4bb4-b101-0a9c3fe9ecd3-kube-api-access-qz6xn\" (UniqueName: \"kubernetes.io/projected/6fd276ef-522b-4bb4-b101-0a9c3fe9ecd3-kube-api-access-qz6xn\") pod \"6fd276ef-522b-4bb4-b101-0a9c3fe9ecd3\" (UID: \"6fd276ef-522b-4bb4-b101-0a9c3fe9ecd3\") "
Oct 30 22:40:30.565053 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:30.564863    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/951a6f7e-a22e-40f2-ace2-1ce5d3bc9b49-home" pod "951a6f7e-a22e-40f2-ace2-1ce5d3bc9b49" (UID: "951a6f7e-a22e-40f2-ace2-1ce5d3bc9b49"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:40:30.565053 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:30.564870    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fd276ef-522b-4bb4-b101-0a9c3fe9ecd3-tmp-dir" pod "6fd276ef-522b-4bb4-b101-0a9c3fe9ecd3" (UID: "6fd276ef-522b-4bb4-b101-0a9c3fe9ecd3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:40:30.565053 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:30.564883    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/951a6f7e-a22e-40f2-ace2-1ce5d3bc9b49-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/951a6f7e-a22e-40f2-ace2-1ce5d3bc9b49-model-cache\") pod \"951a6f7e-a22e-40f2-ace2-1ce5d3bc9b49\" (UID: \"951a6f7e-a22e-40f2-ace2-1ce5d3bc9b49\") "
Oct 30 22:40:30.565053 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:30.564945    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/951a6f7e-a22e-40f2-ace2-1ce5d3bc9b49-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/951a6f7e-a22e-40f2-ace2-1ce5d3bc9b49-tmp-dir\") pod \"951a6f7e-a22e-40f2-ace2-1ce5d3bc9b49\" (UID: \"951a6f7e-a22e-40f2-ace2-1ce5d3bc9b49\") "
Oct 30 22:40:30.565533 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:30.565071    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fd276ef-522b-4bb4-b101-0a9c3fe9ecd3-model-cache" pod "6fd276ef-522b-4bb4-b101-0a9c3fe9ecd3" (UID: "6fd276ef-522b-4bb4-b101-0a9c3fe9ecd3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:40:30.565533 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:30.565213    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fd276ef-522b-4bb4-b101-0a9c3fe9ecd3-home" pod "6fd276ef-522b-4bb4-b101-0a9c3fe9ecd3" (UID: "6fd276ef-522b-4bb4-b101-0a9c3fe9ecd3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:40:30.565533 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:30.565450    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/951a6f7e-a22e-40f2-ace2-1ce5d3bc9b49-tmp-dir" pod "951a6f7e-a22e-40f2-ace2-1ce5d3bc9b49" (UID: "951a6f7e-a22e-40f2-ace2-1ce5d3bc9b49"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:40:30.565533 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:30.565471    5678 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6fd276ef-522b-4bb4-b101-0a9c3fe9ecd3-model-cache\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:30.565533 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:30.565493    5678 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6fd276ef-522b-4bb4-b101-0a9c3fe9ecd3-home\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:30.565533 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:30.565508    5678 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/951a6f7e-a22e-40f2-ace2-1ce5d3bc9b49-home\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:30.565533 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:30.565520    5678 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6fd276ef-522b-4bb4-b101-0a9c3fe9ecd3-tmp-dir\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:30.565936 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:30.565593    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/951a6f7e-a22e-40f2-ace2-1ce5d3bc9b49-model-cache" pod "951a6f7e-a22e-40f2-ace2-1ce5d3bc9b49" (UID: "951a6f7e-a22e-40f2-ace2-1ce5d3bc9b49"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:40:30.568112 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:30.567371    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fd276ef-522b-4bb4-b101-0a9c3fe9ecd3-dshm" pod "6fd276ef-522b-4bb4-b101-0a9c3fe9ecd3" (UID: "6fd276ef-522b-4bb4-b101-0a9c3fe9ecd3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:40:30.568112 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:30.567376    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/951a6f7e-a22e-40f2-ace2-1ce5d3bc9b49-kube-api-access-jmpvb" pod "951a6f7e-a22e-40f2-ace2-1ce5d3bc9b49" (UID: "951a6f7e-a22e-40f2-ace2-1ce5d3bc9b49"). InnerVolumeSpecName "kube-api-access-jmpvb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:40:30.568112 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:30.567533    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6fd276ef-522b-4bb4-b101-0a9c3fe9ecd3-tls-certs" pod "6fd276ef-522b-4bb4-b101-0a9c3fe9ecd3" (UID: "6fd276ef-522b-4bb4-b101-0a9c3fe9ecd3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:40:30.568112 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:30.567553    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/951a6f7e-a22e-40f2-ace2-1ce5d3bc9b49-tls-certs" pod "951a6f7e-a22e-40f2-ace2-1ce5d3bc9b49" (UID: "951a6f7e-a22e-40f2-ace2-1ce5d3bc9b49"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:40:30.568112 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:30.567790    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fd276ef-522b-4bb4-b101-0a9c3fe9ecd3-kube-api-access-qz6xn" pod "6fd276ef-522b-4bb4-b101-0a9c3fe9ecd3" (UID: "6fd276ef-522b-4bb4-b101-0a9c3fe9ecd3"). InnerVolumeSpecName "kube-api-access-qz6xn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:40:30.568112 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:30.567885    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/951a6f7e-a22e-40f2-ace2-1ce5d3bc9b49-dshm" pod "951a6f7e-a22e-40f2-ace2-1ce5d3bc9b49" (UID: "951a6f7e-a22e-40f2-ace2-1ce5d3bc9b49"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:40:30.666957 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:30.666876    5678 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/951a6f7e-a22e-40f2-ace2-1ce5d3bc9b49-tls-certs\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:30.666957 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:30.666903    5678 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6fd276ef-522b-4bb4-b101-0a9c3fe9ecd3-tls-certs\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:30.666957 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:30.666913    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qz6xn\" (UniqueName: \"kubernetes.io/projected/6fd276ef-522b-4bb4-b101-0a9c3fe9ecd3-kube-api-access-qz6xn\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:30.666957 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:30.666923    5678 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/951a6f7e-a22e-40f2-ace2-1ce5d3bc9b49-model-cache\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:30.666957 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:30.666934    5678 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/951a6f7e-a22e-40f2-ace2-1ce5d3bc9b49-tmp-dir\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:30.666957 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:30.666944    5678 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/951a6f7e-a22e-40f2-ace2-1ce5d3bc9b49-dshm\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:30.666957 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:30.666952    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmpvb\" (UniqueName: \"kubernetes.io/projected/951a6f7e-a22e-40f2-ace2-1ce5d3bc9b49-kube-api-access-jmpvb\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:30.666957 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:30.666960    5678 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6fd276ef-522b-4bb4-b101-0a9c3fe9ecd3-dshm\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:30.712485 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:30.712448    5678 generic.go:356] "Generic (PLEG): container finished" podID="805b2e79-a950-415f-8f98-9c01061d0479" containerID="60c78f33c65112c090858ae6419d3003b9b82d40344881d3a2700fb5570ddc59" exitCode=0
Oct 30 22:40:30.712958 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:30.712533    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a00d9b-hj4wb" event={"ID":"805b2e79-a950-415f-8f98-9c01061d0479","Type":"ContainerDied","Data":"60c78f33c65112c090858ae6419d3003b9b82d40344881d3a2700fb5570ddc59"}
Oct 30 22:40:30.713908 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:30.713884    5678 generic.go:356] "Generic (PLEG): container finished" podID="951a6f7e-a22e-40f2-ace2-1ce5d3bc9b49" containerID="e5809108b6519b3887dd567430d21c8ad497ea330ec6fd117ba229c8435fbda5" exitCode=0
Oct 30 22:40:30.714018 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:30.713922    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-fb0b93/simulator-w6-fb0b93-kserve-55b784b479-588bn"
Oct 30 22:40:30.714018 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:30.713962    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-fb0b93/simulator-w6-fb0b93-kserve-55b784b479-588bn" event={"ID":"951a6f7e-a22e-40f2-ace2-1ce5d3bc9b49","Type":"ContainerDied","Data":"e5809108b6519b3887dd567430d21c8ad497ea330ec6fd117ba229c8435fbda5"}
Oct 30 22:40:30.714018 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:30.713993    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-fb0b93/simulator-w6-fb0b93-kserve-55b784b479-588bn" event={"ID":"951a6f7e-a22e-40f2-ace2-1ce5d3bc9b49","Type":"ContainerDied","Data":"ff061666f7b2e8f3da57580f94b59ba143e2cd93523abc06fadb8973aafb27dc"}
Oct 30 22:40:30.714018 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:30.714006    5678 scope.go:122] "RemoveContainer" containerID="e5809108b6519b3887dd567430d21c8ad497ea330ec6fd117ba229c8435fbda5"
Oct 30 22:40:30.715266 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:30.715247    5678 generic.go:356] "Generic (PLEG): container finished" podID="6fd276ef-522b-4bb4-b101-0a9c3fe9ecd3" containerID="8997f6d274894fb2ebc9077cd8197d93f8c588510a2364ea8dc174114d982ed5" exitCode=0
Oct 30 22:40:30.715344 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:30.715307    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-fb0b93/distinct-2-w6-fb0b93-kserve-7d65bb4698-m6tkh" event={"ID":"6fd276ef-522b-4bb4-b101-0a9c3fe9ecd3","Type":"ContainerDied","Data":"8997f6d274894fb2ebc9077cd8197d93f8c588510a2364ea8dc174114d982ed5"}
Oct 30 22:40:30.715344 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:30.715325    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-fb0b93/distinct-2-w6-fb0b93-kserve-7d65bb4698-m6tkh" event={"ID":"6fd276ef-522b-4bb4-b101-0a9c3fe9ecd3","Type":"ContainerDied","Data":"61f8438948d1cfc08745c4cd6defc3cb0c5c78d949b2c300a244ece461463dde"}
Oct 30 22:40:30.715502 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:30.715368    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-fb0b93/distinct-2-w6-fb0b93-kserve-7d65bb4698-m6tkh"
Oct 30 22:40:30.723494 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:30.723477    5678 scope.go:122] "RemoveContainer" containerID="e5809108b6519b3887dd567430d21c8ad497ea330ec6fd117ba229c8435fbda5"
Oct 30 22:40:30.723770 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:40:30.723746    5678 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5809108b6519b3887dd567430d21c8ad497ea330ec6fd117ba229c8435fbda5\": container with ID starting with e5809108b6519b3887dd567430d21c8ad497ea330ec6fd117ba229c8435fbda5 not found: ID does not exist" containerID="e5809108b6519b3887dd567430d21c8ad497ea330ec6fd117ba229c8435fbda5"
Oct 30 22:40:30.723867 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:30.723779    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5809108b6519b3887dd567430d21c8ad497ea330ec6fd117ba229c8435fbda5"} err="failed to get container status \"e5809108b6519b3887dd567430d21c8ad497ea330ec6fd117ba229c8435fbda5\": rpc error: code = NotFound desc = could not find container \"e5809108b6519b3887dd567430d21c8ad497ea330ec6fd117ba229c8435fbda5\": container with ID starting with e5809108b6519b3887dd567430d21c8ad497ea330ec6fd117ba229c8435fbda5 not found: ID does not exist"
Oct 30 22:40:30.723867 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:30.723800    5678 scope.go:122] "RemoveContainer" containerID="8997f6d274894fb2ebc9077cd8197d93f8c588510a2364ea8dc174114d982ed5"
Oct 30 22:40:30.732259 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:30.732243    5678 scope.go:122] "RemoveContainer" containerID="8997f6d274894fb2ebc9077cd8197d93f8c588510a2364ea8dc174114d982ed5"
Oct 30 22:40:30.732570 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:40:30.732554    5678 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8997f6d274894fb2ebc9077cd8197d93f8c588510a2364ea8dc174114d982ed5\": container with ID starting with 8997f6d274894fb2ebc9077cd8197d93f8c588510a2364ea8dc174114d982ed5 not found: ID does not exist" containerID="8997f6d274894fb2ebc9077cd8197d93f8c588510a2364ea8dc174114d982ed5"
Oct 30 22:40:30.732633 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:30.732577    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8997f6d274894fb2ebc9077cd8197d93f8c588510a2364ea8dc174114d982ed5"} err="failed to get container status \"8997f6d274894fb2ebc9077cd8197d93f8c588510a2364ea8dc174114d982ed5\": rpc error: code = NotFound desc = could not find container \"8997f6d274894fb2ebc9077cd8197d93f8c588510a2364ea8dc174114d982ed5\": container with ID starting with 8997f6d274894fb2ebc9077cd8197d93f8c588510a2364ea8dc174114d982ed5 not found: ID does not exist"
Oct 30 22:40:30.928611 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:30.928529    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-fb0b93/simulator-w6-fb0b93-kserve-55b784b479-588bn"]
Oct 30 22:40:30.933686 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:30.933660    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-fb0b93/simulator-w6-fb0b93-kserve-55b784b479-588bn"]
Oct 30 22:40:31.039955 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:31.039923    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-fb0b93/distinct-2-w6-fb0b93-kserve-7d65bb4698-m6tkh"]
Oct 30 22:40:31.046827 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:31.046800    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-fb0b93/distinct-2-w6-fb0b93-kserve-7d65bb4698-m6tkh"]
Oct 30 22:40:31.858652 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:31.858623    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a00d9b-hj4wb"
Oct 30 22:40:31.978561 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:31.978475    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/805b2e79-a950-415f-8f98-9c01061d0479-kube-api-access-c6ncp\" (UniqueName: \"kubernetes.io/projected/805b2e79-a950-415f-8f98-9c01061d0479-kube-api-access-c6ncp\") pod \"805b2e79-a950-415f-8f98-9c01061d0479\" (UID: \"805b2e79-a950-415f-8f98-9c01061d0479\") "
Oct 30 22:40:31.978731 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:31.978573    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/805b2e79-a950-415f-8f98-9c01061d0479-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/805b2e79-a950-415f-8f98-9c01061d0479-maas-api-xks-ca\") pod \"805b2e79-a950-415f-8f98-9c01061d0479\" (UID: \"805b2e79-a950-415f-8f98-9c01061d0479\") "
Oct 30 22:40:31.978731 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:31.978640    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/805b2e79-a950-415f-8f98-9c01061d0479-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/805b2e79-a950-415f-8f98-9c01061d0479-maas-api-openshift-ca\") pod \"805b2e79-a950-415f-8f98-9c01061d0479\" (UID: \"805b2e79-a950-415f-8f98-9c01061d0479\") "
Oct 30 22:40:31.979076 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:31.979048    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/805b2e79-a950-415f-8f98-9c01061d0479-maas-api-openshift-ca" pod "805b2e79-a950-415f-8f98-9c01061d0479" (UID: "805b2e79-a950-415f-8f98-9c01061d0479"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:40:31.980890 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:31.980857    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/805b2e79-a950-415f-8f98-9c01061d0479-kube-api-access-c6ncp" pod "805b2e79-a950-415f-8f98-9c01061d0479" (UID: "805b2e79-a950-415f-8f98-9c01061d0479"). InnerVolumeSpecName "kube-api-access-c6ncp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:40:31.981032 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:31.980959    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/805b2e79-a950-415f-8f98-9c01061d0479-maas-api-xks-ca" pod "805b2e79-a950-415f-8f98-9c01061d0479" (UID: "805b2e79-a950-415f-8f98-9c01061d0479"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:40:32.017718 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:32.017688    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28e87096-54mrw"]
Oct 30 22:40:32.018062 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:32.018050    5678 cpu_manager.go:414] "containerMap: removing container" podUID="805b2e79-a950-415f-8f98-9c01061d0479" containerName="revoke-keys"
Oct 30 22:40:32.018137 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:32.018063    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="805b2e79-a950-415f-8f98-9c01061d0479" containerName="revoke-keys"
Oct 30 22:40:32.018137 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:32.018086    5678 cpu_manager.go:414] "containerMap: removing container" podUID="6fd276ef-522b-4bb4-b101-0a9c3fe9ecd3" containerName="main"
Oct 30 22:40:32.018137 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:32.018091    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fd276ef-522b-4bb4-b101-0a9c3fe9ecd3" containerName="main"
Oct 30 22:40:32.018137 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:32.018104    5678 cpu_manager.go:414] "containerMap: removing container" podUID="e325caa7-d4fb-4a0e-9fe0-ee95742278b3" containerName="istio-proxy"
Oct 30 22:40:32.018137 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:32.018110    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e325caa7-d4fb-4a0e-9fe0-ee95742278b3" containerName="istio-proxy"
Oct 30 22:40:32.018137 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:32.018117    5678 cpu_manager.go:414] "containerMap: removing container" podUID="99f99481-73d0-4fb4-9c88-12ee143a6904" containerName="istio-proxy"
Oct 30 22:40:32.018137 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:32.018122    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99f99481-73d0-4fb4-9c88-12ee143a6904" containerName="istio-proxy"
Oct 30 22:40:32.018137 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:32.018135    5678 cpu_manager.go:414] "containerMap: removing container" podUID="f8eba014-2735-4189-a982-e33413ce8e4b" containerName="istio-proxy"
Oct 30 22:40:32.018137 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:32.018139    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8eba014-2735-4189-a982-e33413ce8e4b" containerName="istio-proxy"
Oct 30 22:40:32.018613 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:32.018151    5678 cpu_manager.go:414] "containerMap: removing container" podUID="951a6f7e-a22e-40f2-ace2-1ce5d3bc9b49" containerName="main"
Oct 30 22:40:32.018613 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:32.018156    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="951a6f7e-a22e-40f2-ace2-1ce5d3bc9b49" containerName="main"
Oct 30 22:40:32.018613 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:32.018210    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="99f99481-73d0-4fb4-9c88-12ee143a6904" containerName="istio-proxy"
Oct 30 22:40:32.018613 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:32.018219    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="951a6f7e-a22e-40f2-ace2-1ce5d3bc9b49" containerName="main"
Oct 30 22:40:32.018613 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:32.018225    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="805b2e79-a950-415f-8f98-9c01061d0479" containerName="revoke-keys"
Oct 30 22:40:32.018613 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:32.018231    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="e325caa7-d4fb-4a0e-9fe0-ee95742278b3" containerName="istio-proxy"
Oct 30 22:40:32.018613 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:32.018238    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fd276ef-522b-4bb4-b101-0a9c3fe9ecd3" containerName="main"
Oct 30 22:40:32.021416 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:32.021381    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e87096-54mrw"
Oct 30 22:40:32.047681 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:32.047639    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28e87096-54mrw"]
Oct 30 22:40:32.080273 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:32.080243    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/805b2e79-a950-415f-8f98-9c01061d0479-maas-api-openshift-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:32.080273 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:32.080268    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6ncp\" (UniqueName: \"kubernetes.io/projected/805b2e79-a950-415f-8f98-9c01061d0479-kube-api-access-c6ncp\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:32.080471 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:32.080282    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/805b2e79-a950-415f-8f98-9c01061d0479-maas-api-xks-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:32.181026 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:32.180987    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e99e89f9-7b36-4876-883d-11432b7aa108-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-28e87096-54mrw\" (UID: \"e99e89f9-7b36-4876-883d-11432b7aa108\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e87096-54mrw"
Oct 30 22:40:32.181199 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:32.181075    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72k5j\" (UniqueName: \"kubernetes.io/projected/e99e89f9-7b36-4876-883d-11432b7aa108-kube-api-access-72k5j\") pod \"maas-api-revoke-sub-28e87096-54mrw\" (UID: \"e99e89f9-7b36-4876-883d-11432b7aa108\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e87096-54mrw"
Oct 30 22:40:32.181199 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:32.181129    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e99e89f9-7b36-4876-883d-11432b7aa108-maas-api-xks-ca\") pod \"maas-api-revoke-sub-28e87096-54mrw\" (UID: \"e99e89f9-7b36-4876-883d-11432b7aa108\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e87096-54mrw"
Oct 30 22:40:32.281883 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:32.281793    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72k5j\" (UniqueName: \"kubernetes.io/projected/e99e89f9-7b36-4876-883d-11432b7aa108-kube-api-access-72k5j\") pod \"maas-api-revoke-sub-28e87096-54mrw\" (UID: \"e99e89f9-7b36-4876-883d-11432b7aa108\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e87096-54mrw"
Oct 30 22:40:32.281883 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:32.281837    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e99e89f9-7b36-4876-883d-11432b7aa108-maas-api-xks-ca\") pod \"maas-api-revoke-sub-28e87096-54mrw\" (UID: \"e99e89f9-7b36-4876-883d-11432b7aa108\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e87096-54mrw"
Oct 30 22:40:32.282122 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:32.281899    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e99e89f9-7b36-4876-883d-11432b7aa108-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-28e87096-54mrw\" (UID: \"e99e89f9-7b36-4876-883d-11432b7aa108\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e87096-54mrw"
Oct 30 22:40:32.282539 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:32.282517    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e99e89f9-7b36-4876-883d-11432b7aa108-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-28e87096-54mrw\" (UID: \"e99e89f9-7b36-4876-883d-11432b7aa108\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e87096-54mrw"
Oct 30 22:40:32.284262 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:32.284242    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e99e89f9-7b36-4876-883d-11432b7aa108-maas-api-xks-ca\") pod \"maas-api-revoke-sub-28e87096-54mrw\" (UID: \"e99e89f9-7b36-4876-883d-11432b7aa108\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e87096-54mrw"
Oct 30 22:40:32.340437 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:32.340394    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72k5j\" (UniqueName: \"kubernetes.io/projected/e99e89f9-7b36-4876-883d-11432b7aa108-kube-api-access-72k5j\") pod \"maas-api-revoke-sub-28e87096-54mrw\" (UID: \"e99e89f9-7b36-4876-883d-11432b7aa108\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e87096-54mrw"
Oct 30 22:40:32.522181 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:32.522146    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6fd276ef-522b-4bb4-b101-0a9c3fe9ecd3" path="/var/lib/kubelet/pods/6fd276ef-522b-4bb4-b101-0a9c3fe9ecd3/volumes"
Oct 30 22:40:32.522711 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:32.522695    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="951a6f7e-a22e-40f2-ace2-1ce5d3bc9b49" path="/var/lib/kubelet/pods/951a6f7e-a22e-40f2-ace2-1ce5d3bc9b49/volumes"
Oct 30 22:40:32.636257 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:32.636227    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e87096-54mrw"
Oct 30 22:40:32.666576 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:32.666545    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-a00d9b/distinct-2-w1-a00d9b-kserve-68fd95957c-dmgnb"]
Oct 30 22:40:32.731165 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:32.731132    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a00d9b-hj4wb"
Oct 30 22:40:32.731165 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:32.731160    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a00d9b-hj4wb" event={"ID":"805b2e79-a950-415f-8f98-9c01061d0479","Type":"ContainerDied","Data":"809f4ec1b802be1b86c5c8061f1ba6fab06848b5361afadd0b43d860429e3332"}
Oct 30 22:40:32.731472 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:32.731200    5678 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="809f4ec1b802be1b86c5c8061f1ba6fab06848b5361afadd0b43d860429e3332"
Oct 30 22:40:33.022691 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:33.022597    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28e87096-54mrw"]
Oct 30 22:40:33.031291 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:40:33.031263    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode99e89f9_7b36_4876_883d_11432b7aa108.slice/crio-3450f87169f3eafb1e269f34218e3c1368370fd15cb95a3306f51a310e954c99 WatchSource:0}: Error finding container 3450f87169f3eafb1e269f34218e3c1368370fd15cb95a3306f51a310e954c99: Status 404 returned error can't find the container with id 3450f87169f3eafb1e269f34218e3c1368370fd15cb95a3306f51a310e954c99
Oct 30 22:40:33.207043 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:33.207000    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-fb0b93-7fb666d8f9-wr46f" podUID="24066223-6a66-43ef-a85d-83183bc37fa7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:40:33.738845 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:33.738816    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e87096-54mrw" event={"ID":"e99e89f9-7b36-4876-883d-11432b7aa108","Type":"ContainerStarted","Data":"da32e171a3f0f4368b920fc6c78407b32d0afeb9afdb479f542781e17d083b63"}
Oct 30 22:40:33.738995 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:33.738866    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e87096-54mrw" event={"ID":"e99e89f9-7b36-4876-883d-11432b7aa108","Type":"ContainerStarted","Data":"3450f87169f3eafb1e269f34218e3c1368370fd15cb95a3306f51a310e954c99"}
Oct 30 22:40:33.855042 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:33.852278    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e87096-54mrw" podStartSLOduration=2.852259854 podStartE2EDuration="2.852259854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:40:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:40:33.849198342 +0000 UTC m=+3011.453505913" watchObservedRunningTime="2026-10-30 22:40:33.852259854 +0000 UTC m=+3011.456567410"
Oct 30 22:40:34.746193 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:34.746155    5678 generic.go:356] "Generic (PLEG): container finished" podID="e99e89f9-7b36-4876-883d-11432b7aa108" containerID="da32e171a3f0f4368b920fc6c78407b32d0afeb9afdb479f542781e17d083b63" exitCode=0
Oct 30 22:40:34.746606 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:34.746242    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e87096-54mrw" event={"ID":"e99e89f9-7b36-4876-883d-11432b7aa108","Type":"ContainerDied","Data":"da32e171a3f0f4368b920fc6c78407b32d0afeb9afdb479f542781e17d083b63"}
Oct 30 22:40:35.875526 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:35.875500    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e87096-54mrw"
Oct 30 22:40:36.017581 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:36.017497    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e99e89f9-7b36-4876-883d-11432b7aa108-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e99e89f9-7b36-4876-883d-11432b7aa108-maas-api-openshift-ca\") pod \"e99e89f9-7b36-4876-883d-11432b7aa108\" (UID: \"e99e89f9-7b36-4876-883d-11432b7aa108\") "
Oct 30 22:40:36.017743 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:36.017581    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e99e89f9-7b36-4876-883d-11432b7aa108-kube-api-access-72k5j\" (UniqueName: \"kubernetes.io/projected/e99e89f9-7b36-4876-883d-11432b7aa108-kube-api-access-72k5j\") pod \"e99e89f9-7b36-4876-883d-11432b7aa108\" (UID: \"e99e89f9-7b36-4876-883d-11432b7aa108\") "
Oct 30 22:40:36.017743 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:36.017620    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e99e89f9-7b36-4876-883d-11432b7aa108-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e99e89f9-7b36-4876-883d-11432b7aa108-maas-api-xks-ca\") pod \"e99e89f9-7b36-4876-883d-11432b7aa108\" (UID: \"e99e89f9-7b36-4876-883d-11432b7aa108\") "
Oct 30 22:40:36.017930 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:36.017906    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e99e89f9-7b36-4876-883d-11432b7aa108-maas-api-openshift-ca" pod "e99e89f9-7b36-4876-883d-11432b7aa108" (UID: "e99e89f9-7b36-4876-883d-11432b7aa108"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:40:36.018023 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:36.018006    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e99e89f9-7b36-4876-883d-11432b7aa108-maas-api-openshift-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:36.019999 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:36.019945    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e99e89f9-7b36-4876-883d-11432b7aa108-maas-api-xks-ca" pod "e99e89f9-7b36-4876-883d-11432b7aa108" (UID: "e99e89f9-7b36-4876-883d-11432b7aa108"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:40:36.020523 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:36.020482    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e99e89f9-7b36-4876-883d-11432b7aa108-kube-api-access-72k5j" pod "e99e89f9-7b36-4876-883d-11432b7aa108" (UID: "e99e89f9-7b36-4876-883d-11432b7aa108"). InnerVolumeSpecName "kube-api-access-72k5j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:40:36.119063 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:36.119037    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72k5j\" (UniqueName: \"kubernetes.io/projected/e99e89f9-7b36-4876-883d-11432b7aa108-kube-api-access-72k5j\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:36.119063 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:36.119062    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e99e89f9-7b36-4876-883d-11432b7aa108-maas-api-xks-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:36.758394 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:36.758363    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e87096-54mrw"
Oct 30 22:40:36.758394 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:36.758381    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e87096-54mrw" event={"ID":"e99e89f9-7b36-4876-883d-11432b7aa108","Type":"ContainerDied","Data":"3450f87169f3eafb1e269f34218e3c1368370fd15cb95a3306f51a310e954c99"}
Oct 30 22:40:36.758620 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:36.758449    5678 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3450f87169f3eafb1e269f34218e3c1368370fd15cb95a3306f51a310e954c99"
Oct 30 22:40:38.730994 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:38.730953    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-a00d9b-55d8cbb68c-dvsfc"]
Oct 30 22:40:40.571490 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:40:40.571452    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 22:40:41.700574 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:41.700537    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14ada201-cmcmd"]
Oct 30 22:40:41.700967 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:41.700913    5678 cpu_manager.go:414] "containerMap: removing container" podUID="e99e89f9-7b36-4876-883d-11432b7aa108" containerName="revoke-keys"
Oct 30 22:40:41.700967 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:41.700923    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e99e89f9-7b36-4876-883d-11432b7aa108" containerName="revoke-keys"
Oct 30 22:40:41.701043 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:41.700996    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="e99e89f9-7b36-4876-883d-11432b7aa108" containerName="revoke-keys"
Oct 30 22:40:41.704132 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:41.704113    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14ada201-cmcmd"
Oct 30 22:40:41.759675 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:41.759646    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14ada201-cmcmd"]
Oct 30 22:40:41.774713 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:41.774686    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63984586-39b2-4924-96eb-f0890de5ed03-maas-api-xks-ca\") pod \"maas-api-revoke-sub-14ada201-cmcmd\" (UID: \"63984586-39b2-4924-96eb-f0890de5ed03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14ada201-cmcmd"
Oct 30 22:40:41.774869 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:41.774809    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ld7zl\" (UniqueName: \"kubernetes.io/projected/63984586-39b2-4924-96eb-f0890de5ed03-kube-api-access-ld7zl\") pod \"maas-api-revoke-sub-14ada201-cmcmd\" (UID: \"63984586-39b2-4924-96eb-f0890de5ed03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14ada201-cmcmd"
Oct 30 22:40:41.774979 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:41.774951    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63984586-39b2-4924-96eb-f0890de5ed03-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-14ada201-cmcmd\" (UID: \"63984586-39b2-4924-96eb-f0890de5ed03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14ada201-cmcmd"
Oct 30 22:40:41.804178 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:41.804143    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:40:41.804178 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:41.804158    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:40:41.804443 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:41.804197    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:40:41.804443 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:41.804270    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5rvcj\""
Oct 30 22:40:41.876385 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:41.876347    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ld7zl\" (UniqueName: \"kubernetes.io/projected/63984586-39b2-4924-96eb-f0890de5ed03-kube-api-access-ld7zl\") pod \"maas-api-revoke-sub-14ada201-cmcmd\" (UID: \"63984586-39b2-4924-96eb-f0890de5ed03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14ada201-cmcmd"
Oct 30 22:40:41.876574 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:41.876420    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63984586-39b2-4924-96eb-f0890de5ed03-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-14ada201-cmcmd\" (UID: \"63984586-39b2-4924-96eb-f0890de5ed03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14ada201-cmcmd"
Oct 30 22:40:41.876574 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:41.876444    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63984586-39b2-4924-96eb-f0890de5ed03-maas-api-xks-ca\") pod \"maas-api-revoke-sub-14ada201-cmcmd\" (UID: \"63984586-39b2-4924-96eb-f0890de5ed03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14ada201-cmcmd"
Oct 30 22:40:41.877063 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:41.877039    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63984586-39b2-4924-96eb-f0890de5ed03-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-14ada201-cmcmd\" (UID: \"63984586-39b2-4924-96eb-f0890de5ed03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14ada201-cmcmd"
Oct 30 22:40:41.878693 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:41.878676    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63984586-39b2-4924-96eb-f0890de5ed03-maas-api-xks-ca\") pod \"maas-api-revoke-sub-14ada201-cmcmd\" (UID: \"63984586-39b2-4924-96eb-f0890de5ed03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14ada201-cmcmd"
Oct 30 22:40:41.935725 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:41.935697    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ld7zl\" (UniqueName: \"kubernetes.io/projected/63984586-39b2-4924-96eb-f0890de5ed03-kube-api-access-ld7zl\") pod \"maas-api-revoke-sub-14ada201-cmcmd\" (UID: \"63984586-39b2-4924-96eb-f0890de5ed03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14ada201-cmcmd"
Oct 30 22:40:42.017165 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:42.017086    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14ada201-cmcmd"
Oct 30 22:40:42.396431 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:42.396385    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14ada201-cmcmd"]
Oct 30 22:40:42.404012 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:40:42.403984    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63984586_39b2_4924_96eb_f0890de5ed03.slice/crio-6417c2641c10f23010c328886789e9eef67138abcc614779ef92f44f87b2e9b5 WatchSource:0}: Error finding container 6417c2641c10f23010c328886789e9eef67138abcc614779ef92f44f87b2e9b5: Status 404 returned error can't find the container with id 6417c2641c10f23010c328886789e9eef67138abcc614779ef92f44f87b2e9b5
Oct 30 22:40:42.791261 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:42.791233    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14ada201-cmcmd" event={"ID":"63984586-39b2-4924-96eb-f0890de5ed03","Type":"ContainerStarted","Data":"4504d45f1e8b356ed49218e41c9482c91399a1fffe65b821389939a763c60889"}
Oct 30 22:40:42.791618 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:42.791283    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14ada201-cmcmd" event={"ID":"63984586-39b2-4924-96eb-f0890de5ed03","Type":"ContainerStarted","Data":"6417c2641c10f23010c328886789e9eef67138abcc614779ef92f44f87b2e9b5"}
Oct 30 22:40:42.904204 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:42.904161    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14ada201-cmcmd" podStartSLOduration=1.904147295 podStartE2EDuration="1.904147295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:40:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:40:42.902469775 +0000 UTC m=+3020.506777328" watchObservedRunningTime="2026-10-30 22:40:42.904147295 +0000 UTC m=+3020.508454849"
Oct 30 22:40:43.207374 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:43.207339    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-fb0b93-7fb666d8f9-wr46f" podUID="24066223-6a66-43ef-a85d-83183bc37fa7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:40:43.732038 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:43.732000    5678 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a00d9b-55d8cbb68c-dvsfc" podUID="a2f7629d-e9a0-438b-b4b0-b33d5b6a47cb" containerName="payload-pre-processing" containerID="cri-o://215376cb1f32dd4a31c29cc482b7bd8680821c0cff40a6089f3fc7c2c9354150" gracePeriod=25
Oct 30 22:40:43.798190 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:43.798155    5678 generic.go:356] "Generic (PLEG): container finished" podID="63984586-39b2-4924-96eb-f0890de5ed03" containerID="4504d45f1e8b356ed49218e41c9482c91399a1fffe65b821389939a763c60889" exitCode=0
Oct 30 22:40:43.798579 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:43.798202    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14ada201-cmcmd" event={"ID":"63984586-39b2-4924-96eb-f0890de5ed03","Type":"ContainerDied","Data":"4504d45f1e8b356ed49218e41c9482c91399a1fffe65b821389939a763c60889"}
Oct 30 22:40:44.081576 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:44.081489    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a00d9b-55d8cbb68c-dvsfc" podUID="a2f7629d-e9a0-438b-b4b0-b33d5b6a47cb" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:40:44.935652 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:44.935623    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14ada201-cmcmd"
Oct 30 22:40:45.005186 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:45.005151    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63984586-39b2-4924-96eb-f0890de5ed03-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63984586-39b2-4924-96eb-f0890de5ed03-maas-api-xks-ca\") pod \"63984586-39b2-4924-96eb-f0890de5ed03\" (UID: \"63984586-39b2-4924-96eb-f0890de5ed03\") "
Oct 30 22:40:45.005325 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:45.005236    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63984586-39b2-4924-96eb-f0890de5ed03-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63984586-39b2-4924-96eb-f0890de5ed03-maas-api-openshift-ca\") pod \"63984586-39b2-4924-96eb-f0890de5ed03\" (UID: \"63984586-39b2-4924-96eb-f0890de5ed03\") "
Oct 30 22:40:45.005325 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:45.005292    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63984586-39b2-4924-96eb-f0890de5ed03-kube-api-access-ld7zl\" (UniqueName: \"kubernetes.io/projected/63984586-39b2-4924-96eb-f0890de5ed03-kube-api-access-ld7zl\") pod \"63984586-39b2-4924-96eb-f0890de5ed03\" (UID: \"63984586-39b2-4924-96eb-f0890de5ed03\") "
Oct 30 22:40:45.005574 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:45.005547    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63984586-39b2-4924-96eb-f0890de5ed03-maas-api-openshift-ca" pod "63984586-39b2-4924-96eb-f0890de5ed03" (UID: "63984586-39b2-4924-96eb-f0890de5ed03"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:40:45.005687 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:45.005635    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63984586-39b2-4924-96eb-f0890de5ed03-maas-api-openshift-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:45.007213 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:45.007183    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63984586-39b2-4924-96eb-f0890de5ed03-maas-api-xks-ca" pod "63984586-39b2-4924-96eb-f0890de5ed03" (UID: "63984586-39b2-4924-96eb-f0890de5ed03"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:40:45.007293 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:45.007241    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63984586-39b2-4924-96eb-f0890de5ed03-kube-api-access-ld7zl" pod "63984586-39b2-4924-96eb-f0890de5ed03" (UID: "63984586-39b2-4924-96eb-f0890de5ed03"). InnerVolumeSpecName "kube-api-access-ld7zl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:40:45.106454 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:45.106376    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63984586-39b2-4924-96eb-f0890de5ed03-maas-api-xks-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:45.106454 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:45.106423    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ld7zl\" (UniqueName: \"kubernetes.io/projected/63984586-39b2-4924-96eb-f0890de5ed03-kube-api-access-ld7zl\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:45.808757 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:45.808722    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14ada201-cmcmd" event={"ID":"63984586-39b2-4924-96eb-f0890de5ed03","Type":"ContainerDied","Data":"6417c2641c10f23010c328886789e9eef67138abcc614779ef92f44f87b2e9b5"}
Oct 30 22:40:45.808950 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:45.808775    5678 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6417c2641c10f23010c328886789e9eef67138abcc614779ef92f44f87b2e9b5"
Oct 30 22:40:45.809191 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:45.809154    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14ada201-cmcmd"
Oct 30 22:40:46.379643 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:46.379618    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-fb0b93-7fb666d8f9-wr46f"
Oct 30 22:40:46.423750 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:46.423683    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24066223-6a66-43ef-a85d-83183bc37fa7-kube-api-access-tdrcd\" (UniqueName: \"kubernetes.io/projected/24066223-6a66-43ef-a85d-83183bc37fa7-kube-api-access-tdrcd\") pod \"24066223-6a66-43ef-a85d-83183bc37fa7\" (UID: \"24066223-6a66-43ef-a85d-83183bc37fa7\") "
Oct 30 22:40:46.423750 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:46.423726    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24066223-6a66-43ef-a85d-83183bc37fa7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24066223-6a66-43ef-a85d-83183bc37fa7-plugins-config-volume\") pod \"24066223-6a66-43ef-a85d-83183bc37fa7\" (UID: \"24066223-6a66-43ef-a85d-83183bc37fa7\") "
Oct 30 22:40:46.424155 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:46.424127    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24066223-6a66-43ef-a85d-83183bc37fa7-plugins-config-volume" pod "24066223-6a66-43ef-a85d-83183bc37fa7" (UID: "24066223-6a66-43ef-a85d-83183bc37fa7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:40:46.425818 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:46.425793    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24066223-6a66-43ef-a85d-83183bc37fa7-kube-api-access-tdrcd" pod "24066223-6a66-43ef-a85d-83183bc37fa7" (UID: "24066223-6a66-43ef-a85d-83183bc37fa7"). InnerVolumeSpecName "kube-api-access-tdrcd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:40:46.524731 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:46.524697    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdrcd\" (UniqueName: \"kubernetes.io/projected/24066223-6a66-43ef-a85d-83183bc37fa7-kube-api-access-tdrcd\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:46.524731 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:46.524727    5678 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24066223-6a66-43ef-a85d-83183bc37fa7-plugins-config-volume\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:46.816091 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:46.816006    5678 generic.go:356] "Generic (PLEG): container finished" podID="24066223-6a66-43ef-a85d-83183bc37fa7" containerID="caef6a84c7871725c67e240f5e7d0a2b96ce07698edb2d00fd3566e1e8e78a98" exitCode=0
Oct 30 22:40:46.816091 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:46.816046    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-fb0b93-7fb666d8f9-wr46f" event={"ID":"24066223-6a66-43ef-a85d-83183bc37fa7","Type":"ContainerDied","Data":"caef6a84c7871725c67e240f5e7d0a2b96ce07698edb2d00fd3566e1e8e78a98"}
Oct 30 22:40:46.816091 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:46.816053    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-fb0b93-7fb666d8f9-wr46f"
Oct 30 22:40:46.816091 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:46.816086    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-fb0b93-7fb666d8f9-wr46f" event={"ID":"24066223-6a66-43ef-a85d-83183bc37fa7","Type":"ContainerDied","Data":"fd252ce6030cbcce33bb5e0774214bdd6564de3a784bd5796a81524bf47d9507"}
Oct 30 22:40:46.816445 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:46.816109    5678 scope.go:122] "RemoveContainer" containerID="caef6a84c7871725c67e240f5e7d0a2b96ce07698edb2d00fd3566e1e8e78a98"
Oct 30 22:40:46.824698 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:46.824683    5678 scope.go:122] "RemoveContainer" containerID="caef6a84c7871725c67e240f5e7d0a2b96ce07698edb2d00fd3566e1e8e78a98"
Oct 30 22:40:46.824946 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:40:46.824929    5678 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"caef6a84c7871725c67e240f5e7d0a2b96ce07698edb2d00fd3566e1e8e78a98\": container with ID starting with caef6a84c7871725c67e240f5e7d0a2b96ce07698edb2d00fd3566e1e8e78a98 not found: ID does not exist" containerID="caef6a84c7871725c67e240f5e7d0a2b96ce07698edb2d00fd3566e1e8e78a98"
Oct 30 22:40:46.825016 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:46.824955    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"caef6a84c7871725c67e240f5e7d0a2b96ce07698edb2d00fd3566e1e8e78a98"} err="failed to get container status \"caef6a84c7871725c67e240f5e7d0a2b96ce07698edb2d00fd3566e1e8e78a98\": rpc error: code = NotFound desc = could not find container \"caef6a84c7871725c67e240f5e7d0a2b96ce07698edb2d00fd3566e1e8e78a98\": container with ID starting with caef6a84c7871725c67e240f5e7d0a2b96ce07698edb2d00fd3566e1e8e78a98 not found: ID does not exist"
Oct 30 22:40:46.972633 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:46.972598    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-fb0b93-7fb666d8f9-wr46f"]
Oct 30 22:40:46.977756 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:46.977729    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-fb0b93-7fb666d8f9-wr46f"]
Oct 30 22:40:47.687532 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:47.687493    5678 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-a00d9b/distinct-2-w1-a00d9b-kserve-68fd95957c-dmgnb" podUID="23a81fda-f7e3-4f91-8d96-9f19c7de6e09" containerName="main" containerID="cri-o://11b87afbda21db67b69a33372ded4712e8f6fc805e169106936e32e8f2070d2a" gracePeriod=45
Oct 30 22:40:47.824189 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:47.824160    5678 generic.go:356] "Generic (PLEG): container finished" podID="23a81fda-f7e3-4f91-8d96-9f19c7de6e09" containerID="11b87afbda21db67b69a33372ded4712e8f6fc805e169106936e32e8f2070d2a" exitCode=0
Oct 30 22:40:47.824332 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:47.824237    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a00d9b/distinct-2-w1-a00d9b-kserve-68fd95957c-dmgnb" event={"ID":"23a81fda-f7e3-4f91-8d96-9f19c7de6e09","Type":"ContainerDied","Data":"11b87afbda21db67b69a33372ded4712e8f6fc805e169106936e32e8f2070d2a"}
Oct 30 22:40:47.933515 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:47.933494    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a00d9b/distinct-2-w1-a00d9b-kserve-68fd95957c-dmgnb"
Oct 30 22:40:48.039687 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:48.039614    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23a81fda-f7e3-4f91-8d96-9f19c7de6e09-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23a81fda-f7e3-4f91-8d96-9f19c7de6e09-tmp-dir\") pod \"23a81fda-f7e3-4f91-8d96-9f19c7de6e09\" (UID: \"23a81fda-f7e3-4f91-8d96-9f19c7de6e09\") "
Oct 30 22:40:48.039687 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:48.039674    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/23a81fda-f7e3-4f91-8d96-9f19c7de6e09-tls-certs\" (UniqueName: \"kubernetes.io/secret/23a81fda-f7e3-4f91-8d96-9f19c7de6e09-tls-certs\") pod \"23a81fda-f7e3-4f91-8d96-9f19c7de6e09\" (UID: \"23a81fda-f7e3-4f91-8d96-9f19c7de6e09\") "
Oct 30 22:40:48.039905 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:48.039712    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23a81fda-f7e3-4f91-8d96-9f19c7de6e09-dshm\" (UniqueName: \"kubernetes.io/empty-dir/23a81fda-f7e3-4f91-8d96-9f19c7de6e09-dshm\") pod \"23a81fda-f7e3-4f91-8d96-9f19c7de6e09\" (UID: \"23a81fda-f7e3-4f91-8d96-9f19c7de6e09\") "
Oct 30 22:40:48.039905 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:48.039753    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23a81fda-f7e3-4f91-8d96-9f19c7de6e09-home\" (UniqueName: \"kubernetes.io/empty-dir/23a81fda-f7e3-4f91-8d96-9f19c7de6e09-home\") pod \"23a81fda-f7e3-4f91-8d96-9f19c7de6e09\" (UID: \"23a81fda-f7e3-4f91-8d96-9f19c7de6e09\") "
Oct 30 22:40:48.039905 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:48.039892    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23a81fda-f7e3-4f91-8d96-9f19c7de6e09-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23a81fda-f7e3-4f91-8d96-9f19c7de6e09-model-cache\") pod \"23a81fda-f7e3-4f91-8d96-9f19c7de6e09\" (UID: \"23a81fda-f7e3-4f91-8d96-9f19c7de6e09\") "
Oct 30 22:40:48.040066 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:48.039902    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23a81fda-f7e3-4f91-8d96-9f19c7de6e09-tmp-dir" pod "23a81fda-f7e3-4f91-8d96-9f19c7de6e09" (UID: "23a81fda-f7e3-4f91-8d96-9f19c7de6e09"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:40:48.040066 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:48.039944    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23a81fda-f7e3-4f91-8d96-9f19c7de6e09-kube-api-access-gh2zc\" (UniqueName: \"kubernetes.io/projected/23a81fda-f7e3-4f91-8d96-9f19c7de6e09-kube-api-access-gh2zc\") pod \"23a81fda-f7e3-4f91-8d96-9f19c7de6e09\" (UID: \"23a81fda-f7e3-4f91-8d96-9f19c7de6e09\") "
Oct 30 22:40:48.040066 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:48.039998    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23a81fda-f7e3-4f91-8d96-9f19c7de6e09-home" pod "23a81fda-f7e3-4f91-8d96-9f19c7de6e09" (UID: "23a81fda-f7e3-4f91-8d96-9f19c7de6e09"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:40:48.040237 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:48.040107    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23a81fda-f7e3-4f91-8d96-9f19c7de6e09-model-cache" pod "23a81fda-f7e3-4f91-8d96-9f19c7de6e09" (UID: "23a81fda-f7e3-4f91-8d96-9f19c7de6e09"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:40:48.040323 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:48.040304    5678 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23a81fda-f7e3-4f91-8d96-9f19c7de6e09-tmp-dir\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:48.040373 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:48.040329    5678 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23a81fda-f7e3-4f91-8d96-9f19c7de6e09-home\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:48.040373 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:48.040344    5678 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23a81fda-f7e3-4f91-8d96-9f19c7de6e09-model-cache\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:48.041911 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:48.041885    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23a81fda-f7e3-4f91-8d96-9f19c7de6e09-dshm" pod "23a81fda-f7e3-4f91-8d96-9f19c7de6e09" (UID: "23a81fda-f7e3-4f91-8d96-9f19c7de6e09"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:40:48.042037 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:48.041934    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23a81fda-f7e3-4f91-8d96-9f19c7de6e09-kube-api-access-gh2zc" pod "23a81fda-f7e3-4f91-8d96-9f19c7de6e09" (UID: "23a81fda-f7e3-4f91-8d96-9f19c7de6e09"). InnerVolumeSpecName "kube-api-access-gh2zc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:40:48.042037 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:48.041982    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23a81fda-f7e3-4f91-8d96-9f19c7de6e09-tls-certs" pod "23a81fda-f7e3-4f91-8d96-9f19c7de6e09" (UID: "23a81fda-f7e3-4f91-8d96-9f19c7de6e09"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:40:48.141732 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:48.141702    5678 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23a81fda-f7e3-4f91-8d96-9f19c7de6e09-dshm\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:48.141732 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:48.141731    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gh2zc\" (UniqueName: \"kubernetes.io/projected/23a81fda-f7e3-4f91-8d96-9f19c7de6e09-kube-api-access-gh2zc\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:48.141922 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:48.141742    5678 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23a81fda-f7e3-4f91-8d96-9f19c7de6e09-tls-certs\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:48.522152 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:48.522118    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24066223-6a66-43ef-a85d-83183bc37fa7" path="/var/lib/kubelet/pods/24066223-6a66-43ef-a85d-83183bc37fa7/volumes"
Oct 30 22:40:48.831324 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:48.831252    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a00d9b/distinct-2-w1-a00d9b-kserve-68fd95957c-dmgnb"
Oct 30 22:40:48.831324 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:48.831261    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a00d9b/distinct-2-w1-a00d9b-kserve-68fd95957c-dmgnb" event={"ID":"23a81fda-f7e3-4f91-8d96-9f19c7de6e09","Type":"ContainerDied","Data":"301b15bf374e098a338e2f8494763acf3d54740ed0066a7f3bf425fd6ba866f5"}
Oct 30 22:40:48.831324 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:48.831314    5678 scope.go:122] "RemoveContainer" containerID="11b87afbda21db67b69a33372ded4712e8f6fc805e169106936e32e8f2070d2a"
Oct 30 22:40:48.987815 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:48.987790    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-a00d9b/distinct-2-w1-a00d9b-kserve-68fd95957c-dmgnb"]
Oct 30 22:40:49.034883 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:49.034856    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-a00d9b/distinct-2-w1-a00d9b-kserve-68fd95957c-dmgnb"]
Oct 30 22:40:50.521037 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:50.521006    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23a81fda-f7e3-4f91-8d96-9f19c7de6e09" path="/var/lib/kubelet/pods/23a81fda-f7e3-4f91-8d96-9f19c7de6e09/volumes"
Oct 30 22:40:54.081152 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:54.081112    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a00d9b-55d8cbb68c-dvsfc" podUID="a2f7629d-e9a0-438b-b4b0-b33d5b6a47cb" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:40:57.488593 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:57.488557    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a00d9b-hj4wb"]
Oct 30 22:40:57.535294 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:57.535264    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a00d9b-hj4wb"]
Oct 30 22:40:58.521232 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:40:58.521200    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="805b2e79-a950-415f-8f98-9c01061d0479" path="/var/lib/kubelet/pods/805b2e79-a950-415f-8f98-9c01061d0479/volumes"
Oct 30 22:41:03.909196 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:41:03.909159    5678 generic.go:356] "Generic (PLEG): container finished" podID="a2f7629d-e9a0-438b-b4b0-b33d5b6a47cb" containerID="215376cb1f32dd4a31c29cc482b7bd8680821c0cff40a6089f3fc7c2c9354150" exitCode=0
Oct 30 22:41:03.909603 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:41:03.909319    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a00d9b-55d8cbb68c-dvsfc" event={"ID":"a2f7629d-e9a0-438b-b4b0-b33d5b6a47cb","Type":"ContainerDied","Data":"215376cb1f32dd4a31c29cc482b7bd8680821c0cff40a6089f3fc7c2c9354150"}
Oct 30 22:41:03.975084 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:41:03.975057    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a00d9b-55d8cbb68c-dvsfc"
Oct 30 22:41:04.087604 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:41:04.087513    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2f7629d-e9a0-438b-b4b0-b33d5b6a47cb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2f7629d-e9a0-438b-b4b0-b33d5b6a47cb-plugins-config-volume\") pod \"a2f7629d-e9a0-438b-b4b0-b33d5b6a47cb\" (UID: \"a2f7629d-e9a0-438b-b4b0-b33d5b6a47cb\") "
Oct 30 22:41:04.087775 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:41:04.087676    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2f7629d-e9a0-438b-b4b0-b33d5b6a47cb-kube-api-access-vn99n\" (UniqueName: \"kubernetes.io/projected/a2f7629d-e9a0-438b-b4b0-b33d5b6a47cb-kube-api-access-vn99n\") pod \"a2f7629d-e9a0-438b-b4b0-b33d5b6a47cb\" (UID: \"a2f7629d-e9a0-438b-b4b0-b33d5b6a47cb\") "
Oct 30 22:41:04.088135 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:41:04.088103    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2f7629d-e9a0-438b-b4b0-b33d5b6a47cb-plugins-config-volume" pod "a2f7629d-e9a0-438b-b4b0-b33d5b6a47cb" (UID: "a2f7629d-e9a0-438b-b4b0-b33d5b6a47cb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:41:04.090242 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:41:04.090216    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2f7629d-e9a0-438b-b4b0-b33d5b6a47cb-kube-api-access-vn99n" pod "a2f7629d-e9a0-438b-b4b0-b33d5b6a47cb" (UID: "a2f7629d-e9a0-438b-b4b0-b33d5b6a47cb"). InnerVolumeSpecName "kube-api-access-vn99n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:41:04.163289 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:41:04.163254    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9fln8"]
Oct 30 22:41:04.163529 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:41:04.163502    5678 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9fln8" podUID="dbbbcda0-3462-4c73-af87-c1c8fcca4706" containerName="istio-proxy" containerID="cri-o://ad3f73b59bc268bcada7aa7e7d195c6fb763b274faaa710c01d4bcffa3852d6e" gracePeriod=30
Oct 30 22:41:04.188551 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:41:04.188489    5678 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2f7629d-e9a0-438b-b4b0-b33d5b6a47cb-plugins-config-volume\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:41:04.188551 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:41:04.188524    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vn99n\" (UniqueName: \"kubernetes.io/projected/a2f7629d-e9a0-438b-b4b0-b33d5b6a47cb-kube-api-access-vn99n\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:41:04.915552 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:41:04.915525    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a00d9b-55d8cbb68c-dvsfc"
Oct 30 22:41:04.915984 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:41:04.915563    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a00d9b-55d8cbb68c-dvsfc" event={"ID":"a2f7629d-e9a0-438b-b4b0-b33d5b6a47cb","Type":"ContainerDied","Data":"e85d4ae8649d3a77b9562ca67583814273a434c9c60b19ec7223f3394cfcb0d2"}
Oct 30 22:41:04.915984 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:41:04.915624    5678 scope.go:122] "RemoveContainer" containerID="215376cb1f32dd4a31c29cc482b7bd8680821c0cff40a6089f3fc7c2c9354150"
Oct 30 22:41:05.079720 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:41:05.079678    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-a00d9b-55d8cbb68c-dvsfc"]
Oct 30 22:41:05.095634 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:41:05.095602    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-a00d9b-55d8cbb68c-dvsfc"]
Oct 30 22:41:06.521974 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:41:06.521929    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2f7629d-e9a0-438b-b4b0-b33d5b6a47cb" path="/var/lib/kubelet/pods/a2f7629d-e9a0-438b-b4b0-b33d5b6a47cb/volumes"
Oct 30 22:41:07.012475 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:41:07.012437    5678 patch_prober.go:29] interesting pod/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9fln8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:41:07.012725 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:41:07.012496    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9fln8" podUID="dbbbcda0-3462-4c73-af87-c1c8fcca4706" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:41:09.405556 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:41:09.405530    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9fln8"
Oct 30 22:41:09.537556 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:41:09.537481    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbbbcda0-3462-4c73-af87-c1c8fcca4706-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbbbcda0-3462-4c73-af87-c1c8fcca4706-istio-envoy\") pod \"dbbbcda0-3462-4c73-af87-c1c8fcca4706\" (UID: \"dbbbcda0-3462-4c73-af87-c1c8fcca4706\") "
Oct 30 22:41:09.537556 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:41:09.537531    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbbbcda0-3462-4c73-af87-c1c8fcca4706-kube-api-access-jc8zs\" (UniqueName: \"kubernetes.io/projected/dbbbcda0-3462-4c73-af87-c1c8fcca4706-kube-api-access-jc8zs\") pod \"dbbbcda0-3462-4c73-af87-c1c8fcca4706\" (UID: \"dbbbcda0-3462-4c73-af87-c1c8fcca4706\") "
Oct 30 22:41:09.537556 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:41:09.537553    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbbbcda0-3462-4c73-af87-c1c8fcca4706-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbbbcda0-3462-4c73-af87-c1c8fcca4706-istiod-ca-cert\") pod \"dbbbcda0-3462-4c73-af87-c1c8fcca4706\" (UID: \"dbbbcda0-3462-4c73-af87-c1c8fcca4706\") "
Oct 30 22:41:09.537923 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:41:09.537581    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbbbcda0-3462-4c73-af87-c1c8fcca4706-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbbbcda0-3462-4c73-af87-c1c8fcca4706-istio-data\") pod \"dbbbcda0-3462-4c73-af87-c1c8fcca4706\" (UID: \"dbbbcda0-3462-4c73-af87-c1c8fcca4706\") "
Oct 30 22:41:09.537923 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:41:09.537612    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbbbcda0-3462-4c73-af87-c1c8fcca4706-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbbbcda0-3462-4c73-af87-c1c8fcca4706-credential-socket\") pod \"dbbbcda0-3462-4c73-af87-c1c8fcca4706\" (UID: \"dbbbcda0-3462-4c73-af87-c1c8fcca4706\") "
Oct 30 22:41:09.537923 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:41:09.537641    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbbbcda0-3462-4c73-af87-c1c8fcca4706-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbbbcda0-3462-4c73-af87-c1c8fcca4706-workload-socket\") pod \"dbbbcda0-3462-4c73-af87-c1c8fcca4706\" (UID: \"dbbbcda0-3462-4c73-af87-c1c8fcca4706\") "
Oct 30 22:41:09.537923 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:41:09.537676    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbbbcda0-3462-4c73-af87-c1c8fcca4706-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbbbcda0-3462-4c73-af87-c1c8fcca4706-workload-certs\") pod \"dbbbcda0-3462-4c73-af87-c1c8fcca4706\" (UID: \"dbbbcda0-3462-4c73-af87-c1c8fcca4706\") "
Oct 30 22:41:09.537923 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:41:09.537704    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbbbcda0-3462-4c73-af87-c1c8fcca4706-istio-token\" (UniqueName: \"kubernetes.io/projected/dbbbcda0-3462-4c73-af87-c1c8fcca4706-istio-token\") pod \"dbbbcda0-3462-4c73-af87-c1c8fcca4706\" (UID: \"dbbbcda0-3462-4c73-af87-c1c8fcca4706\") "
Oct 30 22:41:09.537923 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:41:09.537737    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dbbbcda0-3462-4c73-af87-c1c8fcca4706-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbbbcda0-3462-4c73-af87-c1c8fcca4706-istio-podinfo\") pod \"dbbbcda0-3462-4c73-af87-c1c8fcca4706\" (UID: \"dbbbcda0-3462-4c73-af87-c1c8fcca4706\") "
Oct 30 22:41:09.538275 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:41:09.537994    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbbbcda0-3462-4c73-af87-c1c8fcca4706-istiod-ca-cert" pod "dbbbcda0-3462-4c73-af87-c1c8fcca4706" (UID: "dbbbcda0-3462-4c73-af87-c1c8fcca4706"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:41:09.538275 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:41:09.538001    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbbbcda0-3462-4c73-af87-c1c8fcca4706-workload-certs" pod "dbbbcda0-3462-4c73-af87-c1c8fcca4706" (UID: "dbbbcda0-3462-4c73-af87-c1c8fcca4706"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:41:09.538275 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:41:09.538239    5678 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbbbcda0-3462-4c73-af87-c1c8fcca4706-istiod-ca-cert\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:41:09.538275 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:41:09.538247    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbbbcda0-3462-4c73-af87-c1c8fcca4706-istio-data" pod "dbbbcda0-3462-4c73-af87-c1c8fcca4706" (UID: "dbbbcda0-3462-4c73-af87-c1c8fcca4706"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:41:09.538529 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:41:09.538259    5678 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbbbcda0-3462-4c73-af87-c1c8fcca4706-workload-certs\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:41:09.538529 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:41:09.538327    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbbbcda0-3462-4c73-af87-c1c8fcca4706-credential-socket" pod "dbbbcda0-3462-4c73-af87-c1c8fcca4706" (UID: "dbbbcda0-3462-4c73-af87-c1c8fcca4706"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:41:09.538529 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:41:09.538392    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbbbcda0-3462-4c73-af87-c1c8fcca4706-workload-socket" pod "dbbbcda0-3462-4c73-af87-c1c8fcca4706" (UID: "dbbbcda0-3462-4c73-af87-c1c8fcca4706"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:41:09.540145 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:41:09.540094    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbbbcda0-3462-4c73-af87-c1c8fcca4706-istio-envoy" pod "dbbbcda0-3462-4c73-af87-c1c8fcca4706" (UID: "dbbbcda0-3462-4c73-af87-c1c8fcca4706"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:41:09.540145 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:41:09.540108    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbbbcda0-3462-4c73-af87-c1c8fcca4706-kube-api-access-jc8zs" pod "dbbbcda0-3462-4c73-af87-c1c8fcca4706" (UID: "dbbbcda0-3462-4c73-af87-c1c8fcca4706"). InnerVolumeSpecName "kube-api-access-jc8zs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:41:09.540469 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:41:09.540442    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dbbbcda0-3462-4c73-af87-c1c8fcca4706-istio-podinfo" pod "dbbbcda0-3462-4c73-af87-c1c8fcca4706" (UID: "dbbbcda0-3462-4c73-af87-c1c8fcca4706"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:41:09.540674 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:41:09.540653    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbbbcda0-3462-4c73-af87-c1c8fcca4706-istio-token" pod "dbbbcda0-3462-4c73-af87-c1c8fcca4706" (UID: "dbbbcda0-3462-4c73-af87-c1c8fcca4706"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:41:09.639277 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:41:09.639249    5678 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbbbcda0-3462-4c73-af87-c1c8fcca4706-istio-envoy\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:41:09.639277 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:41:09.639276    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jc8zs\" (UniqueName: \"kubernetes.io/projected/dbbbcda0-3462-4c73-af87-c1c8fcca4706-kube-api-access-jc8zs\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:41:09.639514 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:41:09.639286    5678 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbbbcda0-3462-4c73-af87-c1c8fcca4706-istio-data\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:41:09.639514 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:41:09.639296    5678 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbbbcda0-3462-4c73-af87-c1c8fcca4706-credential-socket\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:41:09.639514 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:41:09.639306    5678 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbbbcda0-3462-4c73-af87-c1c8fcca4706-workload-socket\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:41:09.639514 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:41:09.639321    5678 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbbbcda0-3462-4c73-af87-c1c8fcca4706-istio-token\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:41:09.639514 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:41:09.639330    5678 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbbbcda0-3462-4c73-af87-c1c8fcca4706-istio-podinfo\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:41:09.943328 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:41:09.943295    5678 generic.go:356] "Generic (PLEG): container finished" podID="dbbbcda0-3462-4c73-af87-c1c8fcca4706" containerID="ad3f73b59bc268bcada7aa7e7d195c6fb763b274faaa710c01d4bcffa3852d6e" exitCode=0
Oct 30 22:41:09.943508 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:41:09.943345    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9fln8"
Oct 30 22:41:09.943508 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:41:09.943381    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9fln8" event={"ID":"dbbbcda0-3462-4c73-af87-c1c8fcca4706","Type":"ContainerDied","Data":"ad3f73b59bc268bcada7aa7e7d195c6fb763b274faaa710c01d4bcffa3852d6e"}
Oct 30 22:41:09.943508 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:41:09.943439    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9fln8" event={"ID":"dbbbcda0-3462-4c73-af87-c1c8fcca4706","Type":"ContainerDied","Data":"a6eb2f676a5cedcff70bbb53f7b35e1eb4b1e3f8a26e059c68b2c96fdab9df85"}
Oct 30 22:41:09.943508 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:41:09.943463    5678 scope.go:122] "RemoveContainer" containerID="ad3f73b59bc268bcada7aa7e7d195c6fb763b274faaa710c01d4bcffa3852d6e"
Oct 30 22:41:09.953138 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:41:09.953117    5678 scope.go:122] "RemoveContainer" containerID="495ad04d6a89274ae44c70c17d1fc9defd3d924808919743235cb5ea283530fb"
Oct 30 22:41:09.961222 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:41:09.961196    5678 scope.go:122] "RemoveContainer" containerID="ad3f73b59bc268bcada7aa7e7d195c6fb763b274faaa710c01d4bcffa3852d6e"
Oct 30 22:41:09.961524 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:41:09.961503    5678 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad3f73b59bc268bcada7aa7e7d195c6fb763b274faaa710c01d4bcffa3852d6e\": container with ID starting with ad3f73b59bc268bcada7aa7e7d195c6fb763b274faaa710c01d4bcffa3852d6e not found: ID does not exist" containerID="ad3f73b59bc268bcada7aa7e7d195c6fb763b274faaa710c01d4bcffa3852d6e"
Oct 30 22:41:09.961585 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:41:09.961535    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad3f73b59bc268bcada7aa7e7d195c6fb763b274faaa710c01d4bcffa3852d6e"} err="failed to get container status \"ad3f73b59bc268bcada7aa7e7d195c6fb763b274faaa710c01d4bcffa3852d6e\": rpc error: code = NotFound desc = could not find container \"ad3f73b59bc268bcada7aa7e7d195c6fb763b274faaa710c01d4bcffa3852d6e\": container with ID starting with ad3f73b59bc268bcada7aa7e7d195c6fb763b274faaa710c01d4bcffa3852d6e not found: ID does not exist"
Oct 30 22:41:09.961585 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:41:09.961555    5678 scope.go:122] "RemoveContainer" containerID="495ad04d6a89274ae44c70c17d1fc9defd3d924808919743235cb5ea283530fb"
Oct 30 22:41:09.961785 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:41:09.961767    5678 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"495ad04d6a89274ae44c70c17d1fc9defd3d924808919743235cb5ea283530fb\": container with ID starting with 495ad04d6a89274ae44c70c17d1fc9defd3d924808919743235cb5ea283530fb not found: ID does not exist" containerID="495ad04d6a89274ae44c70c17d1fc9defd3d924808919743235cb5ea283530fb"
Oct 30 22:41:09.961827 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:41:09.961794    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"495ad04d6a89274ae44c70c17d1fc9defd3d924808919743235cb5ea283530fb"} err="failed to get container status \"495ad04d6a89274ae44c70c17d1fc9defd3d924808919743235cb5ea283530fb\": rpc error: code = NotFound desc = could not find container \"495ad04d6a89274ae44c70c17d1fc9defd3d924808919743235cb5ea283530fb\": container with ID starting with 495ad04d6a89274ae44c70c17d1fc9defd3d924808919743235cb5ea283530fb not found: ID does not exist"
Oct 30 22:41:10.139189 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:41:10.139154    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9fln8"]
Oct 30 22:41:10.185983 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:41:10.185952    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-9fln8"]
Oct 30 22:41:10.522168 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:41:10.522134    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbbbcda0-3462-4c73-af87-c1c8fcca4706" path="/var/lib/kubelet/pods/dbbbcda0-3462-4c73-af87-c1c8fcca4706/volumes"
Oct 30 22:41:10.571804 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:41:10.571772    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 22:41:32.319016 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:41:32.318936    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-wvmfj"]
Oct 30 22:41:32.319390 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:41:32.319320    5678 cpu_manager.go:414] "containerMap: removing container" podUID="24066223-6a66-43ef-a85d-83183bc37fa7" containerName="payload-pre-processing"
Oct 30 22:41:32.319390 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:41:32.319332    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24066223-6a66-43ef-a85d-83183bc37fa7" containerName="payload-pre-processing"
Oct 30 22:41:32.319390 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:41:32.319351    5678 cpu_manager.go:414] "containerMap: removing container" podUID="dbbbcda0-3462-4c73-af87-c1c8fcca4706" containerName="istio-proxy"
Oct 30 22:41:32.319390 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:41:32.319358    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbbbcda0-3462-4c73-af87-c1c8fcca4706" containerName="istio-proxy"
Oct 30 22:41:32.319390 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:41:32.319368    5678 cpu_manager.go:414] "containerMap: removing container" podUID="23a81fda-f7e3-4f91-8d96-9f19c7de6e09" containerName="main"
Oct 30 22:41:32.319390 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:41:32.319372    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23a81fda-f7e3-4f91-8d96-9f19c7de6e09" containerName="main"
Oct 30 22:41:32.319390 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:41:32.319380    5678 cpu_manager.go:414] "containerMap: removing container" podUID="dbbbcda0-3462-4c73-af87-c1c8fcca4706" containerName="istio-proxy"
Oct 30 22:41:32.319390 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:41:32.319384    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbbbcda0-3462-4c73-af87-c1c8fcca4706" containerName="istio-proxy"
Oct 30 22:41:32.319390 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:41:32.319396    5678 cpu_manager.go:414] "containerMap: removing container" podUID="a2f7629d-e9a0-438b-b4b0-b33d5b6a47cb" containerName="payload-pre-processing"
Oct 30 22:41:32.319390 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:41:32.319412    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2f7629d-e9a0-438b-b4b0-b33d5b6a47cb" containerName="payload-pre-processing"
Oct 30 22:41:32.319738 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:41:32.319429    5678 cpu_manager.go:414] "containerMap: removing container" podUID="63984586-39b2-4924-96eb-f0890de5ed03" containerName="revoke-keys"
Oct 30 22:41:32.319738 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:41:32.319436    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63984586-39b2-4924-96eb-f0890de5ed03" containerName="revoke-keys"
Oct 30 22:41:32.319738 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:41:32.319499    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="63984586-39b2-4924-96eb-f0890de5ed03" containerName="revoke-keys"
Oct 30 22:41:32.319738 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:41:32.319515    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="24066223-6a66-43ef-a85d-83183bc37fa7" containerName="payload-pre-processing"
Oct 30 22:41:32.319738 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:41:32.319523    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbbbcda0-3462-4c73-af87-c1c8fcca4706" containerName="istio-proxy"
Oct 30 22:41:32.319738 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:41:32.319528    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="23a81fda-f7e3-4f91-8d96-9f19c7de6e09" containerName="main"
Oct 30 22:41:32.319738 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:41:32.319534    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbbbcda0-3462-4c73-af87-c1c8fcca4706" containerName="istio-proxy"
Oct 30 22:41:32.319738 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:41:32.319540    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2f7629d-e9a0-438b-b4b0-b33d5b6a47cb" containerName="payload-pre-processing"
Oct 30 22:41:32.324271 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:41:32.324249    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-wvmfj"
Oct 30 22:41:32.375280 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:41:32.375102    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:41:32.375503 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:41:32.375299    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:41:32.377332 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:41:32.377310    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-wvmfj"]
Oct 30 22:41:32.421738 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:41:32.421719    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:41:32.421851 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:41:32.421759    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5rvcj\""
Oct 30 22:41:32.440544 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:41:32.440523    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ac7a623-ad5e-44ab-8db5-5e9a69112e16-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-wvmfj\" (UID: \"5ac7a623-ad5e-44ab-8db5-5e9a69112e16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-wvmfj"
Oct 30 22:41:32.440633 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:41:32.440606    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zcmx\" (UniqueName: \"kubernetes.io/projected/5ac7a623-ad5e-44ab-8db5-5e9a69112e16-kube-api-access-5zcmx\") pod \"maas-api-revoke-sub-85237e45-wvmfj\" (UID: \"5ac7a623-ad5e-44ab-8db5-5e9a69112e16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-wvmfj"
Oct 30 22:41:32.440687 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:41:32.440645    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ac7a623-ad5e-44ab-8db5-5e9a69112e16-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-wvmfj\" (UID: \"5ac7a623-ad5e-44ab-8db5-5e9a69112e16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-wvmfj"
Oct 30 22:41:32.541515 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:41:32.541488    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ac7a623-ad5e-44ab-8db5-5e9a69112e16-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-wvmfj\" (UID: \"5ac7a623-ad5e-44ab-8db5-5e9a69112e16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-wvmfj"
Oct 30 22:41:32.541654 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:41:32.541542    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ac7a623-ad5e-44ab-8db5-5e9a69112e16-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-wvmfj\" (UID: \"5ac7a623-ad5e-44ab-8db5-5e9a69112e16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-wvmfj"
Oct 30 22:41:32.541654 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:41:32.541583    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zcmx\" (UniqueName: \"kubernetes.io/projected/5ac7a623-ad5e-44ab-8db5-5e9a69112e16-kube-api-access-5zcmx\") pod \"maas-api-revoke-sub-85237e45-wvmfj\" (UID: \"5ac7a623-ad5e-44ab-8db5-5e9a69112e16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-wvmfj"
Oct 30 22:41:32.542112 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:41:32.542090    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ac7a623-ad5e-44ab-8db5-5e9a69112e16-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-wvmfj\" (UID: \"5ac7a623-ad5e-44ab-8db5-5e9a69112e16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-wvmfj"
Oct 30 22:41:32.543807 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:41:32.543781    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ac7a623-ad5e-44ab-8db5-5e9a69112e16-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-wvmfj\" (UID: \"5ac7a623-ad5e-44ab-8db5-5e9a69112e16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-wvmfj"
Oct 30 22:41:32.600109 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:41:32.600051    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zcmx\" (UniqueName: \"kubernetes.io/projected/5ac7a623-ad5e-44ab-8db5-5e9a69112e16-kube-api-access-5zcmx\") pod \"maas-api-revoke-sub-85237e45-wvmfj\" (UID: \"5ac7a623-ad5e-44ab-8db5-5e9a69112e16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-wvmfj"
Oct 30 22:41:32.637012 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:41:32.636991    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-wvmfj"
Oct 30 22:41:33.018083 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:41:33.018047    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-wvmfj"]
Oct 30 22:41:33.026253 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:41:33.026221    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ac7a623_ad5e_44ab_8db5_5e9a69112e16.slice/crio-8f96ef3b9ab7c39ea79372cf8369de934bde1adadcaf63f4214e94e831a3fa65 WatchSource:0}: Error finding container 8f96ef3b9ab7c39ea79372cf8369de934bde1adadcaf63f4214e94e831a3fa65: Status 404 returned error can't find the container with id 8f96ef3b9ab7c39ea79372cf8369de934bde1adadcaf63f4214e94e831a3fa65
Oct 30 22:41:33.059321 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:41:33.059298    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-wvmfj" event={"ID":"5ac7a623-ad5e-44ab-8db5-5e9a69112e16","Type":"ContainerStarted","Data":"8f96ef3b9ab7c39ea79372cf8369de934bde1adadcaf63f4214e94e831a3fa65"}
Oct 30 22:41:34.066296 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:41:34.066255    5678 generic.go:356] "Generic (PLEG): container finished" podID="5ac7a623-ad5e-44ab-8db5-5e9a69112e16" containerID="83195f87d44d9899f034267dc33d3de33ebfe7ce9da7695320024dda82b76c7f" exitCode=0
Oct 30 22:41:34.066701 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:41:34.066304    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-wvmfj" event={"ID":"5ac7a623-ad5e-44ab-8db5-5e9a69112e16","Type":"ContainerDied","Data":"83195f87d44d9899f034267dc33d3de33ebfe7ce9da7695320024dda82b76c7f"}
Oct 30 22:41:35.201137 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:41:35.201109    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-wvmfj"
Oct 30 22:41:35.267165 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:41:35.267136    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ac7a623-ad5e-44ab-8db5-5e9a69112e16-kube-api-access-5zcmx\" (UniqueName: \"kubernetes.io/projected/5ac7a623-ad5e-44ab-8db5-5e9a69112e16-kube-api-access-5zcmx\") pod \"5ac7a623-ad5e-44ab-8db5-5e9a69112e16\" (UID: \"5ac7a623-ad5e-44ab-8db5-5e9a69112e16\") "
Oct 30 22:41:35.267334 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:41:35.267183    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ac7a623-ad5e-44ab-8db5-5e9a69112e16-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ac7a623-ad5e-44ab-8db5-5e9a69112e16-maas-api-openshift-ca\") pod \"5ac7a623-ad5e-44ab-8db5-5e9a69112e16\" (UID: \"5ac7a623-ad5e-44ab-8db5-5e9a69112e16\") "
Oct 30 22:41:35.267334 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:41:35.267240    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ac7a623-ad5e-44ab-8db5-5e9a69112e16-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ac7a623-ad5e-44ab-8db5-5e9a69112e16-maas-api-xks-ca\") pod \"5ac7a623-ad5e-44ab-8db5-5e9a69112e16\" (UID: \"5ac7a623-ad5e-44ab-8db5-5e9a69112e16\") "
Oct 30 22:41:35.267613 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:41:35.267587    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ac7a623-ad5e-44ab-8db5-5e9a69112e16-maas-api-openshift-ca" pod "5ac7a623-ad5e-44ab-8db5-5e9a69112e16" (UID: "5ac7a623-ad5e-44ab-8db5-5e9a69112e16"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:41:35.269143 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:41:35.269121    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ac7a623-ad5e-44ab-8db5-5e9a69112e16-kube-api-access-5zcmx" pod "5ac7a623-ad5e-44ab-8db5-5e9a69112e16" (UID: "5ac7a623-ad5e-44ab-8db5-5e9a69112e16"). InnerVolumeSpecName "kube-api-access-5zcmx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:41:35.269242 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:41:35.269178    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ac7a623-ad5e-44ab-8db5-5e9a69112e16-maas-api-xks-ca" pod "5ac7a623-ad5e-44ab-8db5-5e9a69112e16" (UID: "5ac7a623-ad5e-44ab-8db5-5e9a69112e16"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:41:35.368914 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:41:35.368881    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zcmx\" (UniqueName: \"kubernetes.io/projected/5ac7a623-ad5e-44ab-8db5-5e9a69112e16-kube-api-access-5zcmx\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:41:35.368914 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:41:35.368908    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ac7a623-ad5e-44ab-8db5-5e9a69112e16-maas-api-openshift-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:41:35.369102 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:41:35.368920    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ac7a623-ad5e-44ab-8db5-5e9a69112e16-maas-api-xks-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:41:36.076983 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:41:36.076951    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-wvmfj"
Oct 30 22:41:36.077135 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:41:36.077000    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-wvmfj" event={"ID":"5ac7a623-ad5e-44ab-8db5-5e9a69112e16","Type":"ContainerDied","Data":"8f96ef3b9ab7c39ea79372cf8369de934bde1adadcaf63f4214e94e831a3fa65"}
Oct 30 22:41:36.077135 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:41:36.077047    5678 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8f96ef3b9ab7c39ea79372cf8369de934bde1adadcaf63f4214e94e831a3fa65"
Oct 30 22:41:40.574989 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:41:40.574939    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 22:42:10.571056 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:42:10.571019    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 22:42:40.570124 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:42:40.570083    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 22:42:41.225308 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:42:41.225272    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-5rv82"]
Oct 30 22:42:41.225659 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:42:41.225645    5678 cpu_manager.go:414] "containerMap: removing container" podUID="5ac7a623-ad5e-44ab-8db5-5e9a69112e16" containerName="revoke-keys"
Oct 30 22:42:41.225710 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:42:41.225660    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ac7a623-ad5e-44ab-8db5-5e9a69112e16" containerName="revoke-keys"
Oct 30 22:42:41.225746 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:42:41.225741    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ac7a623-ad5e-44ab-8db5-5e9a69112e16" containerName="revoke-keys"
Oct 30 22:42:41.228675 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:42:41.228653    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-5rv82"
Oct 30 22:42:41.279826 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:42:41.279797    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:42:41.279970 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:42:41.279835    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5rvcj\""
Oct 30 22:42:41.280208 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:42:41.280196    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:42:41.326983 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:42:41.326952    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:42:41.329046 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:42:41.329026    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-5rv82"]
Oct 30 22:42:41.352229 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:42:41.352200    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hknwn\" (UniqueName: \"kubernetes.io/projected/e6140d0a-f4e4-4021-ad69-32ea10e389ff-kube-api-access-hknwn\") pod \"maas-api-revoke-sub-c1e271a9-5rv82\" (UID: \"e6140d0a-f4e4-4021-ad69-32ea10e389ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-5rv82"
Oct 30 22:42:41.352352 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:42:41.352243    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6140d0a-f4e4-4021-ad69-32ea10e389ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-5rv82\" (UID: \"e6140d0a-f4e4-4021-ad69-32ea10e389ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-5rv82"
Oct 30 22:42:41.352393 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:42:41.352366    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6140d0a-f4e4-4021-ad69-32ea10e389ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-5rv82\" (UID: \"e6140d0a-f4e4-4021-ad69-32ea10e389ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-5rv82"
Oct 30 22:42:41.453739 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:42:41.453704    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hknwn\" (UniqueName: \"kubernetes.io/projected/e6140d0a-f4e4-4021-ad69-32ea10e389ff-kube-api-access-hknwn\") pod \"maas-api-revoke-sub-c1e271a9-5rv82\" (UID: \"e6140d0a-f4e4-4021-ad69-32ea10e389ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-5rv82"
Oct 30 22:42:41.453935 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:42:41.453750    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6140d0a-f4e4-4021-ad69-32ea10e389ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-5rv82\" (UID: \"e6140d0a-f4e4-4021-ad69-32ea10e389ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-5rv82"
Oct 30 22:42:41.453935 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:42:41.453801    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6140d0a-f4e4-4021-ad69-32ea10e389ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-5rv82\" (UID: \"e6140d0a-f4e4-4021-ad69-32ea10e389ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-5rv82"
Oct 30 22:42:41.454456 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:42:41.454431    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6140d0a-f4e4-4021-ad69-32ea10e389ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-5rv82\" (UID: \"e6140d0a-f4e4-4021-ad69-32ea10e389ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-5rv82"
Oct 30 22:42:41.456153 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:42:41.456129    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6140d0a-f4e4-4021-ad69-32ea10e389ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-5rv82\" (UID: \"e6140d0a-f4e4-4021-ad69-32ea10e389ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-5rv82"
Oct 30 22:42:41.514943 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:42:41.514884    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hknwn\" (UniqueName: \"kubernetes.io/projected/e6140d0a-f4e4-4021-ad69-32ea10e389ff-kube-api-access-hknwn\") pod \"maas-api-revoke-sub-c1e271a9-5rv82\" (UID: \"e6140d0a-f4e4-4021-ad69-32ea10e389ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-5rv82"
Oct 30 22:42:41.541186 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:42:41.541160    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-5rv82"
Oct 30 22:42:41.920068 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:42:41.920040    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-5rv82"]
Oct 30 22:42:41.928084 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:42:41.928056    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6140d0a_f4e4_4021_ad69_32ea10e389ff.slice/crio-6008db4315667dbb9f85606c7c6e95752621d98f7871fcd7cfa33eaaf098e56b WatchSource:0}: Error finding container 6008db4315667dbb9f85606c7c6e95752621d98f7871fcd7cfa33eaaf098e56b: Status 404 returned error can't find the container with id 6008db4315667dbb9f85606c7c6e95752621d98f7871fcd7cfa33eaaf098e56b
Oct 30 22:42:42.397983 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:42:42.397950    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-5rv82" event={"ID":"e6140d0a-f4e4-4021-ad69-32ea10e389ff","Type":"ContainerStarted","Data":"aa34951ec31137690f30f22275e248b6d78cb620bd135c2342371e50905e1125"}
Oct 30 22:42:42.398167 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:42:42.398010    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-5rv82" event={"ID":"e6140d0a-f4e4-4021-ad69-32ea10e389ff","Type":"ContainerStarted","Data":"6008db4315667dbb9f85606c7c6e95752621d98f7871fcd7cfa33eaaf098e56b"}
Oct 30 22:42:42.509394 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:42:42.509302    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-5rv82" podStartSLOduration=1.509289442 podStartE2EDuration="1.509289442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:42:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:42:42.508329994 +0000 UTC m=+3140.112637547" watchObservedRunningTime="2026-10-30 22:42:42.509289442 +0000 UTC m=+3140.113596994"
Oct 30 22:42:43.404062 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:42:43.404024    5678 generic.go:356] "Generic (PLEG): container finished" podID="e6140d0a-f4e4-4021-ad69-32ea10e389ff" containerID="aa34951ec31137690f30f22275e248b6d78cb620bd135c2342371e50905e1125" exitCode=0
Oct 30 22:42:43.404509 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:42:43.404111    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-5rv82" event={"ID":"e6140d0a-f4e4-4021-ad69-32ea10e389ff","Type":"ContainerDied","Data":"aa34951ec31137690f30f22275e248b6d78cb620bd135c2342371e50905e1125"}
Oct 30 22:42:44.542263 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:42:44.542242    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-5rv82"
Oct 30 22:42:44.682507 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:42:44.682438    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6140d0a-f4e4-4021-ad69-32ea10e389ff-kube-api-access-hknwn\" (UniqueName: \"kubernetes.io/projected/e6140d0a-f4e4-4021-ad69-32ea10e389ff-kube-api-access-hknwn\") pod \"e6140d0a-f4e4-4021-ad69-32ea10e389ff\" (UID: \"e6140d0a-f4e4-4021-ad69-32ea10e389ff\") "
Oct 30 22:42:44.682507 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:42:44.682502    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e6140d0a-f4e4-4021-ad69-32ea10e389ff-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6140d0a-f4e4-4021-ad69-32ea10e389ff-maas-api-xks-ca\") pod \"e6140d0a-f4e4-4021-ad69-32ea10e389ff\" (UID: \"e6140d0a-f4e4-4021-ad69-32ea10e389ff\") "
Oct 30 22:42:44.682696 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:42:44.682608    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6140d0a-f4e4-4021-ad69-32ea10e389ff-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6140d0a-f4e4-4021-ad69-32ea10e389ff-maas-api-openshift-ca\") pod \"e6140d0a-f4e4-4021-ad69-32ea10e389ff\" (UID: \"e6140d0a-f4e4-4021-ad69-32ea10e389ff\") "
Oct 30 22:42:44.682980 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:42:44.682948    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6140d0a-f4e4-4021-ad69-32ea10e389ff-maas-api-openshift-ca" pod "e6140d0a-f4e4-4021-ad69-32ea10e389ff" (UID: "e6140d0a-f4e4-4021-ad69-32ea10e389ff"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:42:44.684597 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:42:44.684570    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6140d0a-f4e4-4021-ad69-32ea10e389ff-kube-api-access-hknwn" pod "e6140d0a-f4e4-4021-ad69-32ea10e389ff" (UID: "e6140d0a-f4e4-4021-ad69-32ea10e389ff"). InnerVolumeSpecName "kube-api-access-hknwn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:42:44.684710 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:42:44.684692    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6140d0a-f4e4-4021-ad69-32ea10e389ff-maas-api-xks-ca" pod "e6140d0a-f4e4-4021-ad69-32ea10e389ff" (UID: "e6140d0a-f4e4-4021-ad69-32ea10e389ff"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:42:44.783814 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:42:44.783775    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6140d0a-f4e4-4021-ad69-32ea10e389ff-maas-api-openshift-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:42:44.783814 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:42:44.783809    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hknwn\" (UniqueName: \"kubernetes.io/projected/e6140d0a-f4e4-4021-ad69-32ea10e389ff-kube-api-access-hknwn\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:42:44.784026 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:42:44.783824    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6140d0a-f4e4-4021-ad69-32ea10e389ff-maas-api-xks-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:42:45.415862 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:42:45.415833    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-5rv82"
Oct 30 22:42:45.416040 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:42:45.415833    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-5rv82" event={"ID":"e6140d0a-f4e4-4021-ad69-32ea10e389ff","Type":"ContainerDied","Data":"6008db4315667dbb9f85606c7c6e95752621d98f7871fcd7cfa33eaaf098e56b"}
Oct 30 22:42:45.416040 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:42:45.415968    5678 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6008db4315667dbb9f85606c7c6e95752621d98f7871fcd7cfa33eaaf098e56b"
Oct 30 22:42:56.474800 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:42:56.474768    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-269mj"]
Oct 30 22:42:56.475281 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:42:56.475097    5678 cpu_manager.go:414] "containerMap: removing container" podUID="e6140d0a-f4e4-4021-ad69-32ea10e389ff" containerName="revoke-keys"
Oct 30 22:42:56.475281 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:42:56.475108    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6140d0a-f4e4-4021-ad69-32ea10e389ff" containerName="revoke-keys"
Oct 30 22:42:56.475281 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:42:56.475185    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6140d0a-f4e4-4021-ad69-32ea10e389ff" containerName="revoke-keys"
Oct 30 22:42:56.478278 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:42:56.478257    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-269mj"
Oct 30 22:42:56.521983 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:42:56.521959    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xvtp9"]
Oct 30 22:42:56.525231 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:42:56.525212    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xvtp9"
Oct 30 22:42:56.577410 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:42:56.577373    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-269mj"]
Oct 30 22:42:56.584453 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:42:56.584431    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db21992e-098d-4659-966f-08b506d342dc-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-269mj\" (UID: \"db21992e-098d-4659-966f-08b506d342dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-269mj"
Oct 30 22:42:56.584569 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:42:56.584460    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db21992e-098d-4659-966f-08b506d342dc-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-269mj\" (UID: \"db21992e-098d-4659-966f-08b506d342dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-269mj"
Oct 30 22:42:56.584569 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:42:56.584480    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8194973b-fc01-458a-931f-562b4da099ae-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-xvtp9\" (UID: \"8194973b-fc01-458a-931f-562b4da099ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xvtp9"
Oct 30 22:42:56.584569 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:42:56.584539    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8194973b-fc01-458a-931f-562b4da099ae-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-xvtp9\" (UID: \"8194973b-fc01-458a-931f-562b4da099ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xvtp9"
Oct 30 22:42:56.584686 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:42:56.584633    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8194973b-fc01-458a-931f-562b4da099ae-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-xvtp9\" (UID: \"8194973b-fc01-458a-931f-562b4da099ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xvtp9"
Oct 30 22:42:56.584724 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:42:56.584695    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmpr8\" (UniqueName: \"kubernetes.io/projected/8194973b-fc01-458a-931f-562b4da099ae-kube-api-access-qmpr8\") pod \"maas-default-gateway-openshift-default-85544d4cd7-xvtp9\" (UID: \"8194973b-fc01-458a-931f-562b4da099ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xvtp9"
Oct 30 22:42:56.584724 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:42:56.584713    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db21992e-098d-4659-966f-08b506d342dc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-269mj\" (UID: \"db21992e-098d-4659-966f-08b506d342dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-269mj"
Oct 30 22:42:56.584792 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:42:56.584762    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db21992e-098d-4659-966f-08b506d342dc-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-269mj\" (UID: \"db21992e-098d-4659-966f-08b506d342dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-269mj"
Oct 30 22:42:56.584911 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:42:56.584881    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db21992e-098d-4659-966f-08b506d342dc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-269mj\" (UID: \"db21992e-098d-4659-966f-08b506d342dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-269mj"
Oct 30 22:42:56.585043 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:42:56.584921    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8194973b-fc01-458a-931f-562b4da099ae-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-xvtp9\" (UID: \"8194973b-fc01-458a-931f-562b4da099ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xvtp9"
Oct 30 22:42:56.585043 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:42:56.584991    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmgn6\" (UniqueName: \"kubernetes.io/projected/db21992e-098d-4659-966f-08b506d342dc-kube-api-access-lmgn6\") pod \"maas-default-gateway-openshift-default-85544d4cd7-269mj\" (UID: \"db21992e-098d-4659-966f-08b506d342dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-269mj"
Oct 30 22:42:56.585043 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:42:56.585012    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8194973b-fc01-458a-931f-562b4da099ae-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-xvtp9\" (UID: \"8194973b-fc01-458a-931f-562b4da099ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xvtp9"
Oct 30 22:42:56.585043 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:42:56.585029    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8194973b-fc01-458a-931f-562b4da099ae-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-xvtp9\" (UID: \"8194973b-fc01-458a-931f-562b4da099ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xvtp9"
Oct 30 22:42:56.585290 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:42:56.585047    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db21992e-098d-4659-966f-08b506d342dc-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-269mj\" (UID: \"db21992e-098d-4659-966f-08b506d342dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-269mj"
Oct 30 22:42:56.585290 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:42:56.585202    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db21992e-098d-4659-966f-08b506d342dc-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-269mj\" (UID: \"db21992e-098d-4659-966f-08b506d342dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-269mj"
Oct 30 22:42:56.585290 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:42:56.585262    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8194973b-fc01-458a-931f-562b4da099ae-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-xvtp9\" (UID: \"8194973b-fc01-458a-931f-562b4da099ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xvtp9"
Oct 30 22:42:56.585487 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:42:56.585360    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8194973b-fc01-458a-931f-562b4da099ae-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-xvtp9\" (UID: \"8194973b-fc01-458a-931f-562b4da099ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xvtp9"
Oct 30 22:42:56.585487 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:42:56.585468    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db21992e-098d-4659-966f-08b506d342dc-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-269mj\" (UID: \"db21992e-098d-4659-966f-08b506d342dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-269mj"
Oct 30 22:42:56.625084 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:42:56.625049    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xvtp9"]
Oct 30 22:42:56.686148 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:42:56.686116    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db21992e-098d-4659-966f-08b506d342dc-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-269mj\" (UID: \"db21992e-098d-4659-966f-08b506d342dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-269mj"
Oct 30 22:42:56.686318 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:42:56.686157    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db21992e-098d-4659-966f-08b506d342dc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-269mj\" (UID: \"db21992e-098d-4659-966f-08b506d342dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-269mj"
Oct 30 22:42:56.686318 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:42:56.686177    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8194973b-fc01-458a-931f-562b4da099ae-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-xvtp9\" (UID: \"8194973b-fc01-458a-931f-562b4da099ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xvtp9"
Oct 30 22:42:56.686318 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:42:56.686309    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmgn6\" (UniqueName: \"kubernetes.io/projected/db21992e-098d-4659-966f-08b506d342dc-kube-api-access-lmgn6\") pod \"maas-default-gateway-openshift-default-85544d4cd7-269mj\" (UID: \"db21992e-098d-4659-966f-08b506d342dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-269mj"
Oct 30 22:42:56.686549 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:42:56.686348    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8194973b-fc01-458a-931f-562b4da099ae-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-xvtp9\" (UID: \"8194973b-fc01-458a-931f-562b4da099ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xvtp9"
Oct 30 22:42:56.686549 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:42:56.686378    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8194973b-fc01-458a-931f-562b4da099ae-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-xvtp9\" (UID: \"8194973b-fc01-458a-931f-562b4da099ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xvtp9"
Oct 30 22:42:56.686549 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:42:56.686438    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db21992e-098d-4659-966f-08b506d342dc-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-269mj\" (UID: \"db21992e-098d-4659-966f-08b506d342dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-269mj"
Oct 30 22:42:56.686549 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:42:56.686477    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db21992e-098d-4659-966f-08b506d342dc-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-269mj\" (UID: \"db21992e-098d-4659-966f-08b506d342dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-269mj"
Oct 30 22:42:56.686549 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:42:56.686502    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8194973b-fc01-458a-931f-562b4da099ae-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-xvtp9\" (UID: \"8194973b-fc01-458a-931f-562b4da099ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xvtp9"
Oct 30 22:42:56.686549 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:42:56.686536    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8194973b-fc01-458a-931f-562b4da099ae-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-xvtp9\" (UID: \"8194973b-fc01-458a-931f-562b4da099ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xvtp9"
Oct 30 22:42:56.686861 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:42:56.686592    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8194973b-fc01-458a-931f-562b4da099ae-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-xvtp9\" (UID: \"8194973b-fc01-458a-931f-562b4da099ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xvtp9"
Oct 30 22:42:56.686861 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:42:56.686605    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db21992e-098d-4659-966f-08b506d342dc-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-269mj\" (UID: \"db21992e-098d-4659-966f-08b506d342dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-269mj"
Oct 30 22:42:56.686861 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:42:56.686600    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db21992e-098d-4659-966f-08b506d342dc-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-269mj\" (UID: \"db21992e-098d-4659-966f-08b506d342dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-269mj"
Oct 30 22:42:56.686861 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:42:56.686672    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db21992e-098d-4659-966f-08b506d342dc-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-269mj\" (UID: \"db21992e-098d-4659-966f-08b506d342dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-269mj"
Oct 30 22:42:56.686861 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:42:56.686702    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db21992e-098d-4659-966f-08b506d342dc-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-269mj\" (UID: \"db21992e-098d-4659-966f-08b506d342dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-269mj"
Oct 30 22:42:56.687090 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:42:56.687061    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8194973b-fc01-458a-931f-562b4da099ae-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-xvtp9\" (UID: \"8194973b-fc01-458a-931f-562b4da099ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xvtp9"
Oct 30 22:42:56.687212 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:42:56.687110    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8194973b-fc01-458a-931f-562b4da099ae-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-xvtp9\" (UID: \"8194973b-fc01-458a-931f-562b4da099ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xvtp9"
Oct 30 22:42:56.687212 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:42:56.687142    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8194973b-fc01-458a-931f-562b4da099ae-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-xvtp9\" (UID: \"8194973b-fc01-458a-931f-562b4da099ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xvtp9"
Oct 30 22:42:56.687212 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:42:56.687170    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmpr8\" (UniqueName: \"kubernetes.io/projected/8194973b-fc01-458a-931f-562b4da099ae-kube-api-access-qmpr8\") pod \"maas-default-gateway-openshift-default-85544d4cd7-xvtp9\" (UID: \"8194973b-fc01-458a-931f-562b4da099ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xvtp9"
Oct 30 22:42:56.687212 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:42:56.687194    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db21992e-098d-4659-966f-08b506d342dc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-269mj\" (UID: \"db21992e-098d-4659-966f-08b506d342dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-269mj"
Oct 30 22:42:56.688101 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:42:56.687559    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db21992e-098d-4659-966f-08b506d342dc-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-269mj\" (UID: \"db21992e-098d-4659-966f-08b506d342dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-269mj"
Oct 30 22:42:56.688101 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:42:56.687824    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db21992e-098d-4659-966f-08b506d342dc-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-269mj\" (UID: \"db21992e-098d-4659-966f-08b506d342dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-269mj"
Oct 30 22:42:56.688529 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:42:56.688503    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db21992e-098d-4659-966f-08b506d342dc-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-269mj\" (UID: \"db21992e-098d-4659-966f-08b506d342dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-269mj"
Oct 30 22:42:56.690199 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:42:56.688752    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db21992e-098d-4659-966f-08b506d342dc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-269mj\" (UID: \"db21992e-098d-4659-966f-08b506d342dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-269mj"
Oct 30 22:42:56.690199 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:42:56.688861    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8194973b-fc01-458a-931f-562b4da099ae-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-xvtp9\" (UID: \"8194973b-fc01-458a-931f-562b4da099ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xvtp9"
Oct 30 22:42:56.690199 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:42:56.689259    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db21992e-098d-4659-966f-08b506d342dc-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-269mj\" (UID: \"db21992e-098d-4659-966f-08b506d342dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-269mj"
Oct 30 22:42:56.690199 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:42:56.690015    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8194973b-fc01-458a-931f-562b4da099ae-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-xvtp9\" (UID: \"8194973b-fc01-458a-931f-562b4da099ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xvtp9"
Oct 30 22:42:56.690653 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:42:56.690373    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8194973b-fc01-458a-931f-562b4da099ae-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-xvtp9\" (UID: \"8194973b-fc01-458a-931f-562b4da099ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xvtp9"
Oct 30 22:42:56.690653 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:42:56.690516    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8194973b-fc01-458a-931f-562b4da099ae-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-xvtp9\" (UID: \"8194973b-fc01-458a-931f-562b4da099ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xvtp9"
Oct 30 22:42:56.691523 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:42:56.691475    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db21992e-098d-4659-966f-08b506d342dc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-269mj\" (UID: \"db21992e-098d-4659-966f-08b506d342dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-269mj"
Oct 30 22:42:56.692728 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:42:56.692704    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8194973b-fc01-458a-931f-562b4da099ae-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-xvtp9\" (UID: \"8194973b-fc01-458a-931f-562b4da099ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xvtp9"
Oct 30 22:42:56.693000 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:42:56.691931    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8194973b-fc01-458a-931f-562b4da099ae-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-xvtp9\" (UID: \"8194973b-fc01-458a-931f-562b4da099ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xvtp9"
Oct 30 22:42:56.746194 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:42:56.746131    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8194973b-fc01-458a-931f-562b4da099ae-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-xvtp9\" (UID: \"8194973b-fc01-458a-931f-562b4da099ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xvtp9"
Oct 30 22:42:56.746194 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:42:56.746131    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db21992e-098d-4659-966f-08b506d342dc-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-269mj\" (UID: \"db21992e-098d-4659-966f-08b506d342dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-269mj"
Oct 30 22:42:56.746355 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:42:56.746313    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmgn6\" (UniqueName: \"kubernetes.io/projected/db21992e-098d-4659-966f-08b506d342dc-kube-api-access-lmgn6\") pod \"maas-default-gateway-openshift-default-85544d4cd7-269mj\" (UID: \"db21992e-098d-4659-966f-08b506d342dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-269mj"
Oct 30 22:42:56.746433 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:42:56.746356    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmpr8\" (UniqueName: \"kubernetes.io/projected/8194973b-fc01-458a-931f-562b4da099ae-kube-api-access-qmpr8\") pod \"maas-default-gateway-openshift-default-85544d4cd7-xvtp9\" (UID: \"8194973b-fc01-458a-931f-562b4da099ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xvtp9"
Oct 30 22:42:56.790471 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:42:56.790440    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-269mj"
Oct 30 22:42:56.837596 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:42:56.837543    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xvtp9"
Oct 30 22:42:57.235071 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:42:57.235038    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-269mj"]
Oct 30 22:42:57.237386 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:42:57.237359    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xvtp9"]
Oct 30 22:42:57.242532 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:42:57.242508    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb21992e_098d_4659_966f_08b506d342dc.slice/crio-db6ebdabfdc1da8475b9b32ba99166731fc787f5ddf545e74e715730e8309b3e WatchSource:0}: Error finding container db6ebdabfdc1da8475b9b32ba99166731fc787f5ddf545e74e715730e8309b3e: Status 404 returned error can't find the container with id db6ebdabfdc1da8475b9b32ba99166731fc787f5ddf545e74e715730e8309b3e
Oct 30 22:42:57.244639 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:42:57.244611    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:42:57.244713 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:42:57.244674    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:42:57.244713 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:42:57.244702    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:42:57.300058 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:42:57.300020    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8194973b_fc01_458a_931f_562b4da099ae.slice/crio-33bc06c7b7a102bf337ea64d4156af3d95dfdbfc2c6fdaf479e1107f2c7db1e0 WatchSource:0}: Error finding container 33bc06c7b7a102bf337ea64d4156af3d95dfdbfc2c6fdaf479e1107f2c7db1e0: Status 404 returned error can't find the container with id 33bc06c7b7a102bf337ea64d4156af3d95dfdbfc2c6fdaf479e1107f2c7db1e0
Oct 30 22:42:57.302211 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:42:57.302183    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:42:57.302284 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:42:57.302248    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:42:57.302284 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:42:57.302280    5678 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:42:57.479472 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:42:57.479434    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-269mj" event={"ID":"db21992e-098d-4659-966f-08b506d342dc","Type":"ContainerStarted","Data":"9572aecf2d9a6fe3485bdd9af9c14385519ddc6d2368871737328c33c2d93c87"}
Oct 30 22:42:57.479924 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:42:57.479489    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-269mj" event={"ID":"db21992e-098d-4659-966f-08b506d342dc","Type":"ContainerStarted","Data":"db6ebdabfdc1da8475b9b32ba99166731fc787f5ddf545e74e715730e8309b3e"}
Oct 30 22:42:57.489966 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:42:57.489879    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xvtp9" event={"ID":"8194973b-fc01-458a-931f-562b4da099ae","Type":"ContainerStarted","Data":"65bf4b0175b86dddd76171003c271e3112b5034d81a214284d00bdb1ba1e59df"}
Oct 30 22:42:57.490100 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:42:57.489967    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xvtp9" event={"ID":"8194973b-fc01-458a-931f-562b4da099ae","Type":"ContainerStarted","Data":"33bc06c7b7a102bf337ea64d4156af3d95dfdbfc2c6fdaf479e1107f2c7db1e0"}
Oct 30 22:42:57.598588 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:42:57.598533    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-269mj" podStartSLOduration=1.598518989 podStartE2EDuration="1.598518989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:42:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:42:57.5957828 +0000 UTC m=+3155.200090353" watchObservedRunningTime="2026-10-30 22:42:57.598518989 +0000 UTC m=+3155.202826541"
Oct 30 22:42:57.712209 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:42:57.712153    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xvtp9" podStartSLOduration=1.712136297 podStartE2EDuration="1.712136297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:42:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:42:57.712054339 +0000 UTC m=+3155.316361894" watchObservedRunningTime="2026-10-30 22:42:57.712136297 +0000 UTC m=+3155.316443851"
Oct 30 22:42:57.791791 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:42:57.791663    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-269mj"
Oct 30 22:42:57.838572 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:42:57.838530    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xvtp9"
Oct 30 22:42:58.792254 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:42:58.792213    5678 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-269mj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body=
Oct 30 22:42:58.792749 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:42:58.792275    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-269mj" podUID="db21992e-098d-4659-966f-08b506d342dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused"
Oct 30 22:42:58.852494 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:42:58.852355    5678 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-xvtp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body=
Oct 30 22:42:58.852494 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:42:58.852427    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xvtp9" podUID="8194973b-fc01-458a-931f-562b4da099ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused"
Oct 30 22:42:59.791751 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:42:59.791712    5678 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-269mj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body=
Oct 30 22:42:59.791975 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:42:59.791774    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-269mj" podUID="db21992e-098d-4659-966f-08b506d342dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused"
Oct 30 22:42:59.842851 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:42:59.842812    5678 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-xvtp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body=
Oct 30 22:42:59.843485 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:42:59.843454    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xvtp9" podUID="8194973b-fc01-458a-931f-562b4da099ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused"
Oct 30 22:43:00.791685 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:43:00.791651    5678 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-269mj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body=
Oct 30 22:43:00.792020 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:43:00.791994    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-269mj" podUID="db21992e-098d-4659-966f-08b506d342dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused"
Oct 30 22:43:00.845424 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:43:00.841803    5678 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-xvtp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body=
Oct 30 22:43:00.845424 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:43:00.841871    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xvtp9" podUID="8194973b-fc01-458a-931f-562b4da099ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused"
Oct 30 22:43:01.791958 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:43:01.791862    5678 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-269mj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body=
Oct 30 22:43:01.792290 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:43:01.792244    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-269mj" podUID="db21992e-098d-4659-966f-08b506d342dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused"
Oct 30 22:43:01.841866 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:43:01.841823    5678 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-xvtp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body=
Oct 30 22:43:01.842173 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:43:01.842141    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xvtp9" podUID="8194973b-fc01-458a-931f-562b4da099ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused"
Oct 30 22:43:02.791114 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:43:02.791067    5678 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-269mj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body=
Oct 30 22:43:02.791578 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:43:02.791155    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-269mj" podUID="db21992e-098d-4659-966f-08b506d342dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused"
Oct 30 22:43:02.841892 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:43:02.841849    5678 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-xvtp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body=
Oct 30 22:43:02.842200 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:43:02.842164    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xvtp9" podUID="8194973b-fc01-458a-931f-562b4da099ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused"
Oct 30 22:43:03.791644 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:43:03.791604    5678 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-269mj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body=
Oct 30 22:43:03.792194 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:43:03.791661    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-269mj" podUID="db21992e-098d-4659-966f-08b506d342dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused"
Oct 30 22:43:03.838418 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:43:03.838377    5678 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-xvtp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body=
Oct 30 22:43:03.838583 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:43:03.838453    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xvtp9" podUID="8194973b-fc01-458a-931f-562b4da099ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused"
Oct 30 22:43:04.791284 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:43:04.791243    5678 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-269mj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body=
Oct 30 22:43:04.791482 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:43:04.791310    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-269mj" podUID="db21992e-098d-4659-966f-08b506d342dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused"
Oct 30 22:43:04.837999 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:43:04.837960    5678 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-xvtp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body=
Oct 30 22:43:04.838394 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:43:04.838029    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xvtp9" podUID="8194973b-fc01-458a-931f-562b4da099ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused"
Oct 30 22:43:06.833609 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:43:06.833570    5678 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-269mj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:43:06.834292 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:43:06.834250    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-269mj" podUID="db21992e-098d-4659-966f-08b506d342dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:43:06.878582 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:43:06.878532    5678 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-xvtp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:43:06.878853 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:43:06.878809    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xvtp9" podUID="8194973b-fc01-458a-931f-562b4da099ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:43:07.861494 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:43:07.861455    5678 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-269mj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:43:07.862013 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:43:07.861531    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-269mj" podUID="db21992e-098d-4659-966f-08b506d342dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:43:07.920539 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:43:07.920500    5678 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-xvtp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:43:07.920741 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:43:07.920561    5678 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xvtp9" podUID="8194973b-fc01-458a-931f-562b4da099ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:43:08.058752 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:43:08.058718    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-269mj"
Oct 30 22:43:08.059163 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:43:08.059131    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-269mj"
Oct 30 22:43:08.059854 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:43:08.059831    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-269mj"
Oct 30 22:43:08.062611 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:43:08.062589    5678 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xvtp9"
Oct 30 22:43:08.063142 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:43:08.063119    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xvtp9"
Oct 30 22:43:08.064605 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:43:08.064589    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xvtp9"
Oct 30 22:43:10.570942 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:43:10.570905    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 22:43:12.604279 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:43:12.604250    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-qpfrl"]
Oct 30 22:43:12.607777 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:43:12.607756    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-qpfrl"
Oct 30 22:43:12.659415 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:43:12.659384    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:43:12.706667 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:43:12.706633    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5rvcj\""
Oct 30 22:43:12.706821 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:43:12.706700    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:43:12.708675 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:43:12.708655    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-qpfrl"]
Oct 30 22:43:12.760280 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:43:12.760250    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:43:12.796413 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:43:12.796378    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f95d6797-0031-47cd-acf4-e7a7e41943ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-qpfrl\" (UID: \"f95d6797-0031-47cd-acf4-e7a7e41943ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-qpfrl"
Oct 30 22:43:12.796553 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:43:12.796495    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f95d6797-0031-47cd-acf4-e7a7e41943ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-qpfrl\" (UID: \"f95d6797-0031-47cd-acf4-e7a7e41943ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-qpfrl"
Oct 30 22:43:12.796553 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:43:12.796548    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbd6s\" (UniqueName: \"kubernetes.io/projected/f95d6797-0031-47cd-acf4-e7a7e41943ea-kube-api-access-pbd6s\") pod \"maas-api-revoke-sub-2e1f5b5f-qpfrl\" (UID: \"f95d6797-0031-47cd-acf4-e7a7e41943ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-qpfrl"
Oct 30 22:43:12.897512 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:43:12.897481    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f95d6797-0031-47cd-acf4-e7a7e41943ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-qpfrl\" (UID: \"f95d6797-0031-47cd-acf4-e7a7e41943ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-qpfrl"
Oct 30 22:43:12.897677 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:43:12.897531    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbd6s\" (UniqueName: \"kubernetes.io/projected/f95d6797-0031-47cd-acf4-e7a7e41943ea-kube-api-access-pbd6s\") pod \"maas-api-revoke-sub-2e1f5b5f-qpfrl\" (UID: \"f95d6797-0031-47cd-acf4-e7a7e41943ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-qpfrl"
Oct 30 22:43:12.897677 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:43:12.897578    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f95d6797-0031-47cd-acf4-e7a7e41943ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-qpfrl\" (UID: \"f95d6797-0031-47cd-acf4-e7a7e41943ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-qpfrl"
Oct 30 22:43:12.898124 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:43:12.898099    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f95d6797-0031-47cd-acf4-e7a7e41943ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-qpfrl\" (UID: \"f95d6797-0031-47cd-acf4-e7a7e41943ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-qpfrl"
Oct 30 22:43:12.899787 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:43:12.899767    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f95d6797-0031-47cd-acf4-e7a7e41943ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-qpfrl\" (UID: \"f95d6797-0031-47cd-acf4-e7a7e41943ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-qpfrl"
Oct 30 22:43:12.955880 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:43:12.955853    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbd6s\" (UniqueName: \"kubernetes.io/projected/f95d6797-0031-47cd-acf4-e7a7e41943ea-kube-api-access-pbd6s\") pod \"maas-api-revoke-sub-2e1f5b5f-qpfrl\" (UID: \"f95d6797-0031-47cd-acf4-e7a7e41943ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-qpfrl"
Oct 30 22:43:13.225920 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:43:13.225836    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-qpfrl"
Oct 30 22:43:13.605617 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:43:13.605531    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-qpfrl"]
Oct 30 22:43:13.613059 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:43:13.613031    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf95d6797_0031_47cd_acf4_e7a7e41943ea.slice/crio-baf473bfe66f054d14e3baf5cf14286f81c33989581f6a6de69f1b20467429f6 WatchSource:0}: Error finding container baf473bfe66f054d14e3baf5cf14286f81c33989581f6a6de69f1b20467429f6: Status 404 returned error can't find the container with id baf473bfe66f054d14e3baf5cf14286f81c33989581f6a6de69f1b20467429f6
Oct 30 22:43:14.617829 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:43:14.617793    5678 generic.go:356] "Generic (PLEG): container finished" podID="f95d6797-0031-47cd-acf4-e7a7e41943ea" containerID="e72d11bd3cd5b67cf5eb788f6faffee2d78b913f8ffae9f80fc92956d3cabc4b" exitCode=0
Oct 30 22:43:14.618264 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:43:14.617878    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-qpfrl" event={"ID":"f95d6797-0031-47cd-acf4-e7a7e41943ea","Type":"ContainerDied","Data":"e72d11bd3cd5b67cf5eb788f6faffee2d78b913f8ffae9f80fc92956d3cabc4b"}
Oct 30 22:43:14.618264 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:43:14.617921    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-qpfrl" event={"ID":"f95d6797-0031-47cd-acf4-e7a7e41943ea","Type":"ContainerStarted","Data":"baf473bfe66f054d14e3baf5cf14286f81c33989581f6a6de69f1b20467429f6"}
Oct 30 22:43:15.753689 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:43:15.753665    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-qpfrl"
Oct 30 22:43:15.824893 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:43:15.824866    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f95d6797-0031-47cd-acf4-e7a7e41943ea-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f95d6797-0031-47cd-acf4-e7a7e41943ea-maas-api-openshift-ca\") pod \"f95d6797-0031-47cd-acf4-e7a7e41943ea\" (UID: \"f95d6797-0031-47cd-acf4-e7a7e41943ea\") "
Oct 30 22:43:15.825005 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:43:15.824932    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f95d6797-0031-47cd-acf4-e7a7e41943ea-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f95d6797-0031-47cd-acf4-e7a7e41943ea-maas-api-xks-ca\") pod \"f95d6797-0031-47cd-acf4-e7a7e41943ea\" (UID: \"f95d6797-0031-47cd-acf4-e7a7e41943ea\") "
Oct 30 22:43:15.825084 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:43:15.825022    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f95d6797-0031-47cd-acf4-e7a7e41943ea-kube-api-access-pbd6s\" (UniqueName: \"kubernetes.io/projected/f95d6797-0031-47cd-acf4-e7a7e41943ea-kube-api-access-pbd6s\") pod \"f95d6797-0031-47cd-acf4-e7a7e41943ea\" (UID: \"f95d6797-0031-47cd-acf4-e7a7e41943ea\") "
Oct 30 22:43:15.825214 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:43:15.825192    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f95d6797-0031-47cd-acf4-e7a7e41943ea-maas-api-openshift-ca" pod "f95d6797-0031-47cd-acf4-e7a7e41943ea" (UID: "f95d6797-0031-47cd-acf4-e7a7e41943ea"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:43:15.826947 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:43:15.826913    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f95d6797-0031-47cd-acf4-e7a7e41943ea-maas-api-xks-ca" pod "f95d6797-0031-47cd-acf4-e7a7e41943ea" (UID: "f95d6797-0031-47cd-acf4-e7a7e41943ea"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:43:15.827073 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:43:15.827011    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f95d6797-0031-47cd-acf4-e7a7e41943ea-kube-api-access-pbd6s" pod "f95d6797-0031-47cd-acf4-e7a7e41943ea" (UID: "f95d6797-0031-47cd-acf4-e7a7e41943ea"). InnerVolumeSpecName "kube-api-access-pbd6s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:43:15.926091 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:43:15.926006    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbd6s\" (UniqueName: \"kubernetes.io/projected/f95d6797-0031-47cd-acf4-e7a7e41943ea-kube-api-access-pbd6s\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:43:15.926091 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:43:15.926035    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f95d6797-0031-47cd-acf4-e7a7e41943ea-maas-api-openshift-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:43:15.926091 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:43:15.926048    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f95d6797-0031-47cd-acf4-e7a7e41943ea-maas-api-xks-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:43:16.629625 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:43:16.629594    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-qpfrl"
Oct 30 22:43:16.629796 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:43:16.629629    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-qpfrl" event={"ID":"f95d6797-0031-47cd-acf4-e7a7e41943ea","Type":"ContainerDied","Data":"baf473bfe66f054d14e3baf5cf14286f81c33989581f6a6de69f1b20467429f6"}
Oct 30 22:43:16.629796 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:43:16.629687    5678 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="baf473bfe66f054d14e3baf5cf14286f81c33989581f6a6de69f1b20467429f6"
Oct 30 22:43:25.181673 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:43:25.181639    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-r72qh"]
Oct 30 22:43:25.182076 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:43:25.182008    5678 cpu_manager.go:414] "containerMap: removing container" podUID="f95d6797-0031-47cd-acf4-e7a7e41943ea" containerName="revoke-keys"
Oct 30 22:43:25.182076 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:43:25.182018    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f95d6797-0031-47cd-acf4-e7a7e41943ea" containerName="revoke-keys"
Oct 30 22:43:25.182150 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:43:25.182090    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="f95d6797-0031-47cd-acf4-e7a7e41943ea" containerName="revoke-keys"
Oct 30 22:43:25.185190 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:43:25.185170    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-r72qh"
Oct 30 22:43:25.208154 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:43:25.208125    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c05d3b91-fbb9-43b5-8837-994668e18f51-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-r72qh\" (UID: \"c05d3b91-fbb9-43b5-8837-994668e18f51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-r72qh"
Oct 30 22:43:25.208275 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:43:25.208197    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c05d3b91-fbb9-43b5-8837-994668e18f51-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-r72qh\" (UID: \"c05d3b91-fbb9-43b5-8837-994668e18f51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-r72qh"
Oct 30 22:43:25.208345 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:43:25.208281    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kcb9\" (UniqueName: \"kubernetes.io/projected/c05d3b91-fbb9-43b5-8837-994668e18f51-kube-api-access-9kcb9\") pod \"maas-api-revoke-sub-ec843018-r72qh\" (UID: \"c05d3b91-fbb9-43b5-8837-994668e18f51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-r72qh"
Oct 30 22:43:25.237889 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:43:25.237851    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:43:25.240263 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:43:25.240239    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-r72qh"]
Oct 30 22:43:25.284872 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:43:25.284845    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:43:25.284872 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:43:25.284845    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5rvcj\""
Oct 30 22:43:25.284872 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:43:25.284845    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:43:25.309747 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:43:25.309721    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c05d3b91-fbb9-43b5-8837-994668e18f51-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-r72qh\" (UID: \"c05d3b91-fbb9-43b5-8837-994668e18f51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-r72qh"
Oct 30 22:43:25.309877 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:43:25.309782    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c05d3b91-fbb9-43b5-8837-994668e18f51-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-r72qh\" (UID: \"c05d3b91-fbb9-43b5-8837-994668e18f51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-r72qh"
Oct 30 22:43:25.309877 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:43:25.309822    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kcb9\" (UniqueName: \"kubernetes.io/projected/c05d3b91-fbb9-43b5-8837-994668e18f51-kube-api-access-9kcb9\") pod \"maas-api-revoke-sub-ec843018-r72qh\" (UID: \"c05d3b91-fbb9-43b5-8837-994668e18f51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-r72qh"
Oct 30 22:43:25.310353 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:43:25.310334    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c05d3b91-fbb9-43b5-8837-994668e18f51-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-r72qh\" (UID: \"c05d3b91-fbb9-43b5-8837-994668e18f51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-r72qh"
Oct 30 22:43:25.312106 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:43:25.312083    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c05d3b91-fbb9-43b5-8837-994668e18f51-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-r72qh\" (UID: \"c05d3b91-fbb9-43b5-8837-994668e18f51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-r72qh"
Oct 30 22:43:25.368309 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:43:25.368276    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kcb9\" (UniqueName: \"kubernetes.io/projected/c05d3b91-fbb9-43b5-8837-994668e18f51-kube-api-access-9kcb9\") pod \"maas-api-revoke-sub-ec843018-r72qh\" (UID: \"c05d3b91-fbb9-43b5-8837-994668e18f51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-r72qh"
Oct 30 22:43:25.497569 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:43:25.497499    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-r72qh"
Oct 30 22:43:25.879233 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:43:25.879195    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-r72qh"]
Oct 30 22:43:25.887578 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:43:25.887552    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc05d3b91_fbb9_43b5_8837_994668e18f51.slice/crio-221a9421eec28fd4b16b59a6aa6329907e808d8e4d09bb081c3f9af0d0a0654d WatchSource:0}: Error finding container 221a9421eec28fd4b16b59a6aa6329907e808d8e4d09bb081c3f9af0d0a0654d: Status 404 returned error can't find the container with id 221a9421eec28fd4b16b59a6aa6329907e808d8e4d09bb081c3f9af0d0a0654d
Oct 30 22:43:26.689548 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:43:26.689320    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-r72qh" event={"ID":"c05d3b91-fbb9-43b5-8837-994668e18f51","Type":"ContainerStarted","Data":"09b39c5bbe65326715ff88f78a1310c1a0d26f19bdee1e101824532beb1d40c2"}
Oct 30 22:43:26.689548 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:43:26.689387    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-r72qh" event={"ID":"c05d3b91-fbb9-43b5-8837-994668e18f51","Type":"ContainerStarted","Data":"221a9421eec28fd4b16b59a6aa6329907e808d8e4d09bb081c3f9af0d0a0654d"}
Oct 30 22:43:26.801523 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:43:26.801471    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-r72qh" podStartSLOduration=1.801457503 podStartE2EDuration="1.801457503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:43:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:43:26.800487645 +0000 UTC m=+3184.404795194" watchObservedRunningTime="2026-10-30 22:43:26.801457503 +0000 UTC m=+3184.405765055"
Oct 30 22:43:27.701709 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:43:27.701676    5678 generic.go:356] "Generic (PLEG): container finished" podID="c05d3b91-fbb9-43b5-8837-994668e18f51" containerID="09b39c5bbe65326715ff88f78a1310c1a0d26f19bdee1e101824532beb1d40c2" exitCode=0
Oct 30 22:43:27.702197 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:43:27.701719    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-r72qh" event={"ID":"c05d3b91-fbb9-43b5-8837-994668e18f51","Type":"ContainerDied","Data":"09b39c5bbe65326715ff88f78a1310c1a0d26f19bdee1e101824532beb1d40c2"}
Oct 30 22:43:28.835358 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:43:28.835333    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-r72qh"
Oct 30 22:43:28.944485 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:43:28.944458    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c05d3b91-fbb9-43b5-8837-994668e18f51-kube-api-access-9kcb9\" (UniqueName: \"kubernetes.io/projected/c05d3b91-fbb9-43b5-8837-994668e18f51-kube-api-access-9kcb9\") pod \"c05d3b91-fbb9-43b5-8837-994668e18f51\" (UID: \"c05d3b91-fbb9-43b5-8837-994668e18f51\") "
Oct 30 22:43:28.944649 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:43:28.944511    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c05d3b91-fbb9-43b5-8837-994668e18f51-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c05d3b91-fbb9-43b5-8837-994668e18f51-maas-api-openshift-ca\") pod \"c05d3b91-fbb9-43b5-8837-994668e18f51\" (UID: \"c05d3b91-fbb9-43b5-8837-994668e18f51\") "
Oct 30 22:43:28.944649 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:43:28.944589    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c05d3b91-fbb9-43b5-8837-994668e18f51-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c05d3b91-fbb9-43b5-8837-994668e18f51-maas-api-xks-ca\") pod \"c05d3b91-fbb9-43b5-8837-994668e18f51\" (UID: \"c05d3b91-fbb9-43b5-8837-994668e18f51\") "
Oct 30 22:43:28.944865 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:43:28.944843    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c05d3b91-fbb9-43b5-8837-994668e18f51-maas-api-openshift-ca" pod "c05d3b91-fbb9-43b5-8837-994668e18f51" (UID: "c05d3b91-fbb9-43b5-8837-994668e18f51"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:43:28.946930 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:43:28.946900    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c05d3b91-fbb9-43b5-8837-994668e18f51-maas-api-xks-ca" pod "c05d3b91-fbb9-43b5-8837-994668e18f51" (UID: "c05d3b91-fbb9-43b5-8837-994668e18f51"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:43:28.947241 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:43:28.947213    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c05d3b91-fbb9-43b5-8837-994668e18f51-kube-api-access-9kcb9" pod "c05d3b91-fbb9-43b5-8837-994668e18f51" (UID: "c05d3b91-fbb9-43b5-8837-994668e18f51"). InnerVolumeSpecName "kube-api-access-9kcb9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:43:29.046014 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:43:29.045946    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c05d3b91-fbb9-43b5-8837-994668e18f51-maas-api-xks-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:43:29.046014 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:43:29.045972    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kcb9\" (UniqueName: \"kubernetes.io/projected/c05d3b91-fbb9-43b5-8837-994668e18f51-kube-api-access-9kcb9\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:43:29.046014 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:43:29.045984    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c05d3b91-fbb9-43b5-8837-994668e18f51-maas-api-openshift-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:43:29.711787 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:43:29.711752    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-r72qh" event={"ID":"c05d3b91-fbb9-43b5-8837-994668e18f51","Type":"ContainerDied","Data":"221a9421eec28fd4b16b59a6aa6329907e808d8e4d09bb081c3f9af0d0a0654d"}
Oct 30 22:43:29.711787 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:43:29.711783    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-r72qh"
Oct 30 22:43:29.712026 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:43:29.711801    5678 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="221a9421eec28fd4b16b59a6aa6329907e808d8e4d09bb081c3f9af0d0a0654d"
Oct 30 22:43:40.571104 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:43:40.571065    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 22:44:00.289730 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:44:00.289700    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-f62pp"]
Oct 30 22:44:00.290213 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:44:00.290069    5678 cpu_manager.go:414] "containerMap: removing container" podUID="c05d3b91-fbb9-43b5-8837-994668e18f51" containerName="revoke-keys"
Oct 30 22:44:00.290213 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:44:00.290080    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c05d3b91-fbb9-43b5-8837-994668e18f51" containerName="revoke-keys"
Oct 30 22:44:00.290213 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:44:00.290148    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="c05d3b91-fbb9-43b5-8837-994668e18f51" containerName="revoke-keys"
Oct 30 22:44:00.293263 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:44:00.293243    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-f62pp"
Oct 30 22:44:00.343241 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:44:00.343209    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:44:00.390374 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:44:00.390332    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:44:00.390568 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:44:00.390388    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5rvcj\""
Oct 30 22:44:00.390568 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:44:00.390446    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:44:00.392578 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:44:00.392559    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-f62pp"]
Oct 30 22:44:00.449596 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:44:00.449559    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da43d8a5-64f1-4f97-9841-4f86d7e23cdb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-f62pp\" (UID: \"da43d8a5-64f1-4f97-9841-4f86d7e23cdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-f62pp"
Oct 30 22:44:00.449860 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:44:00.449606    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m46c9\" (UniqueName: \"kubernetes.io/projected/da43d8a5-64f1-4f97-9841-4f86d7e23cdb-kube-api-access-m46c9\") pod \"maas-api-revoke-sub-046e1343-f62pp\" (UID: \"da43d8a5-64f1-4f97-9841-4f86d7e23cdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-f62pp"
Oct 30 22:44:00.449860 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:44:00.449628    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da43d8a5-64f1-4f97-9841-4f86d7e23cdb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-f62pp\" (UID: \"da43d8a5-64f1-4f97-9841-4f86d7e23cdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-f62pp"
Oct 30 22:44:00.550511 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:44:00.550431    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da43d8a5-64f1-4f97-9841-4f86d7e23cdb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-f62pp\" (UID: \"da43d8a5-64f1-4f97-9841-4f86d7e23cdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-f62pp"
Oct 30 22:44:00.550511 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:44:00.550464    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m46c9\" (UniqueName: \"kubernetes.io/projected/da43d8a5-64f1-4f97-9841-4f86d7e23cdb-kube-api-access-m46c9\") pod \"maas-api-revoke-sub-046e1343-f62pp\" (UID: \"da43d8a5-64f1-4f97-9841-4f86d7e23cdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-f62pp"
Oct 30 22:44:00.550511 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:44:00.550483    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da43d8a5-64f1-4f97-9841-4f86d7e23cdb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-f62pp\" (UID: \"da43d8a5-64f1-4f97-9841-4f86d7e23cdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-f62pp"
Oct 30 22:44:00.551094 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:44:00.551075    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da43d8a5-64f1-4f97-9841-4f86d7e23cdb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-f62pp\" (UID: \"da43d8a5-64f1-4f97-9841-4f86d7e23cdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-f62pp"
Oct 30 22:44:00.552734 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:44:00.552715    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da43d8a5-64f1-4f97-9841-4f86d7e23cdb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-f62pp\" (UID: \"da43d8a5-64f1-4f97-9841-4f86d7e23cdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-f62pp"
Oct 30 22:44:00.609535 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:44:00.609505    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m46c9\" (UniqueName: \"kubernetes.io/projected/da43d8a5-64f1-4f97-9841-4f86d7e23cdb-kube-api-access-m46c9\") pod \"maas-api-revoke-sub-046e1343-f62pp\" (UID: \"da43d8a5-64f1-4f97-9841-4f86d7e23cdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-f62pp"
Oct 30 22:44:00.906245 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:44:00.906214    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-f62pp"
Oct 30 22:44:01.291382 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:44:01.291289    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-f62pp"]
Oct 30 22:44:01.299077 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:44:01.299050    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda43d8a5_64f1_4f97_9841_4f86d7e23cdb.slice/crio-08ea37ae362ecdcf47b6cae70bc6c808d18856645661c63a5e5c9a2271e415a1 WatchSource:0}: Error finding container 08ea37ae362ecdcf47b6cae70bc6c808d18856645661c63a5e5c9a2271e415a1: Status 404 returned error can't find the container with id 08ea37ae362ecdcf47b6cae70bc6c808d18856645661c63a5e5c9a2271e415a1
Oct 30 22:44:01.884785 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:44:01.884697    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-f62pp" event={"ID":"da43d8a5-64f1-4f97-9841-4f86d7e23cdb","Type":"ContainerStarted","Data":"9e59d1b00f623a152792b538676049c571f446fced47647487639952bf8d35a7"}
Oct 30 22:44:01.884785 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:44:01.884752    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-f62pp" event={"ID":"da43d8a5-64f1-4f97-9841-4f86d7e23cdb","Type":"ContainerStarted","Data":"08ea37ae362ecdcf47b6cae70bc6c808d18856645661c63a5e5c9a2271e415a1"}
Oct 30 22:44:01.999559 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:44:01.999508    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-f62pp" podStartSLOduration=1.999495664 podStartE2EDuration="1.999495664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:44:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:44:01.997678974 +0000 UTC m=+3219.601986528" watchObservedRunningTime="2026-10-30 22:44:01.999495664 +0000 UTC m=+3219.603803218"
Oct 30 22:44:02.890943 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:44:02.890908    5678 generic.go:356] "Generic (PLEG): container finished" podID="da43d8a5-64f1-4f97-9841-4f86d7e23cdb" containerID="9e59d1b00f623a152792b538676049c571f446fced47647487639952bf8d35a7" exitCode=0
Oct 30 22:44:02.891423 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:44:02.890962    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-f62pp" event={"ID":"da43d8a5-64f1-4f97-9841-4f86d7e23cdb","Type":"ContainerDied","Data":"9e59d1b00f623a152792b538676049c571f446fced47647487639952bf8d35a7"}
Oct 30 22:44:04.048154 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:44:04.048126    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-f62pp"
Oct 30 22:44:04.186520 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:44:04.186443    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da43d8a5-64f1-4f97-9841-4f86d7e23cdb-kube-api-access-m46c9\" (UniqueName: \"kubernetes.io/projected/da43d8a5-64f1-4f97-9841-4f86d7e23cdb-kube-api-access-m46c9\") pod \"da43d8a5-64f1-4f97-9841-4f86d7e23cdb\" (UID: \"da43d8a5-64f1-4f97-9841-4f86d7e23cdb\") "
Oct 30 22:44:04.186520 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:44:04.186479    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da43d8a5-64f1-4f97-9841-4f86d7e23cdb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da43d8a5-64f1-4f97-9841-4f86d7e23cdb-maas-api-xks-ca\") pod \"da43d8a5-64f1-4f97-9841-4f86d7e23cdb\" (UID: \"da43d8a5-64f1-4f97-9841-4f86d7e23cdb\") "
Oct 30 22:44:04.186756 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:44:04.186592    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da43d8a5-64f1-4f97-9841-4f86d7e23cdb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da43d8a5-64f1-4f97-9841-4f86d7e23cdb-maas-api-openshift-ca\") pod \"da43d8a5-64f1-4f97-9841-4f86d7e23cdb\" (UID: \"da43d8a5-64f1-4f97-9841-4f86d7e23cdb\") "
Oct 30 22:44:04.186996 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:44:04.186949    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da43d8a5-64f1-4f97-9841-4f86d7e23cdb-maas-api-openshift-ca" pod "da43d8a5-64f1-4f97-9841-4f86d7e23cdb" (UID: "da43d8a5-64f1-4f97-9841-4f86d7e23cdb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:44:04.188651 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:44:04.188620    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da43d8a5-64f1-4f97-9841-4f86d7e23cdb-maas-api-xks-ca" pod "da43d8a5-64f1-4f97-9841-4f86d7e23cdb" (UID: "da43d8a5-64f1-4f97-9841-4f86d7e23cdb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:44:04.188741 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:44:04.188645    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da43d8a5-64f1-4f97-9841-4f86d7e23cdb-kube-api-access-m46c9" pod "da43d8a5-64f1-4f97-9841-4f86d7e23cdb" (UID: "da43d8a5-64f1-4f97-9841-4f86d7e23cdb"). InnerVolumeSpecName "kube-api-access-m46c9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:44:04.288228 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:44:04.288190    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m46c9\" (UniqueName: \"kubernetes.io/projected/da43d8a5-64f1-4f97-9841-4f86d7e23cdb-kube-api-access-m46c9\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:44:04.288228 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:44:04.288224    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da43d8a5-64f1-4f97-9841-4f86d7e23cdb-maas-api-xks-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:44:04.288228 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:44:04.288234    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da43d8a5-64f1-4f97-9841-4f86d7e23cdb-maas-api-openshift-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:44:04.902486 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:44:04.902455    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-f62pp"
Oct 30 22:44:04.902486 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:44:04.902466    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-f62pp" event={"ID":"da43d8a5-64f1-4f97-9841-4f86d7e23cdb","Type":"ContainerDied","Data":"08ea37ae362ecdcf47b6cae70bc6c808d18856645661c63a5e5c9a2271e415a1"}
Oct 30 22:44:04.902696 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:44:04.902517    5678 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="08ea37ae362ecdcf47b6cae70bc6c808d18856645661c63a5e5c9a2271e415a1"
Oct 30 22:44:10.570598 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:44:10.570564    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 22:44:11.083929 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:44:11.083895    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-2r5j6"]
Oct 30 22:44:11.084294 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:44:11.084279    5678 cpu_manager.go:414] "containerMap: removing container" podUID="da43d8a5-64f1-4f97-9841-4f86d7e23cdb" containerName="revoke-keys"
Oct 30 22:44:11.084345 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:44:11.084296    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da43d8a5-64f1-4f97-9841-4f86d7e23cdb" containerName="revoke-keys"
Oct 30 22:44:11.084390 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:44:11.084381    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="da43d8a5-64f1-4f97-9841-4f86d7e23cdb" containerName="revoke-keys"
Oct 30 22:44:11.087648 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:44:11.087629    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-2r5j6"
Oct 30 22:44:11.152153 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:44:11.152128    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0664869-236d-4245-8d16-374b217dda8d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-2r5j6\" (UID: \"c0664869-236d-4245-8d16-374b217dda8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-2r5j6"
Oct 30 22:44:11.152314 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:44:11.152159    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpqzn\" (UniqueName: \"kubernetes.io/projected/c0664869-236d-4245-8d16-374b217dda8d-kube-api-access-jpqzn\") pod \"maas-api-revoke-sub-d3fba056-2r5j6\" (UID: \"c0664869-236d-4245-8d16-374b217dda8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-2r5j6"
Oct 30 22:44:11.152314 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:44:11.152193    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0664869-236d-4245-8d16-374b217dda8d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-2r5j6\" (UID: \"c0664869-236d-4245-8d16-374b217dda8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-2r5j6"
Oct 30 22:44:11.186864 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:44:11.186832    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:44:11.187008 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:44:11.186862    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:44:11.187008 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:44:11.186894    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5rvcj\""
Oct 30 22:44:11.187008 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:44:11.186941    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:44:11.189127 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:44:11.189107    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-2r5j6"]
Oct 30 22:44:11.252827 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:44:11.252801    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0664869-236d-4245-8d16-374b217dda8d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-2r5j6\" (UID: \"c0664869-236d-4245-8d16-374b217dda8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-2r5j6"
Oct 30 22:44:11.252827 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:44:11.252830    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpqzn\" (UniqueName: \"kubernetes.io/projected/c0664869-236d-4245-8d16-374b217dda8d-kube-api-access-jpqzn\") pod \"maas-api-revoke-sub-d3fba056-2r5j6\" (UID: \"c0664869-236d-4245-8d16-374b217dda8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-2r5j6"
Oct 30 22:44:11.253023 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:44:11.252865    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0664869-236d-4245-8d16-374b217dda8d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-2r5j6\" (UID: \"c0664869-236d-4245-8d16-374b217dda8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-2r5j6"
Oct 30 22:44:11.253449 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:44:11.253432    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0664869-236d-4245-8d16-374b217dda8d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-2r5j6\" (UID: \"c0664869-236d-4245-8d16-374b217dda8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-2r5j6"
Oct 30 22:44:11.255232 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:44:11.255212    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0664869-236d-4245-8d16-374b217dda8d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-2r5j6\" (UID: \"c0664869-236d-4245-8d16-374b217dda8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-2r5j6"
Oct 30 22:44:11.311780 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:44:11.311750    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpqzn\" (UniqueName: \"kubernetes.io/projected/c0664869-236d-4245-8d16-374b217dda8d-kube-api-access-jpqzn\") pod \"maas-api-revoke-sub-d3fba056-2r5j6\" (UID: \"c0664869-236d-4245-8d16-374b217dda8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-2r5j6"
Oct 30 22:44:11.402730 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:44:11.402686    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-2r5j6"
Oct 30 22:44:11.786934 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:44:11.786852    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-2r5j6"]
Oct 30 22:44:11.794605 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:44:11.794578    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0664869_236d_4245_8d16_374b217dda8d.slice/crio-0787647ef3d754ce08bf540a1f68e5cdd66cc02134356edc7b68298f1647ea53 WatchSource:0}: Error finding container 0787647ef3d754ce08bf540a1f68e5cdd66cc02134356edc7b68298f1647ea53: Status 404 returned error can't find the container with id 0787647ef3d754ce08bf540a1f68e5cdd66cc02134356edc7b68298f1647ea53
Oct 30 22:44:11.938882 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:44:11.938851    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-2r5j6" event={"ID":"c0664869-236d-4245-8d16-374b217dda8d","Type":"ContainerStarted","Data":"0787647ef3d754ce08bf540a1f68e5cdd66cc02134356edc7b68298f1647ea53"}
Oct 30 22:44:12.945825 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:44:12.945790    5678 generic.go:356] "Generic (PLEG): container finished" podID="c0664869-236d-4245-8d16-374b217dda8d" containerID="934e0f7a01465f66c6d9c94b7274d0174490086b5a8396b3c99f915971de1f65" exitCode=0
Oct 30 22:44:12.945825 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:44:12.945826    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-2r5j6" event={"ID":"c0664869-236d-4245-8d16-374b217dda8d","Type":"ContainerDied","Data":"934e0f7a01465f66c6d9c94b7274d0174490086b5a8396b3c99f915971de1f65"}
Oct 30 22:44:14.121713 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:44:14.121681    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-2r5j6"
Oct 30 22:44:14.179777 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:44:14.179734    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0664869-236d-4245-8d16-374b217dda8d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0664869-236d-4245-8d16-374b217dda8d-maas-api-xks-ca\") pod \"c0664869-236d-4245-8d16-374b217dda8d\" (UID: \"c0664869-236d-4245-8d16-374b217dda8d\") "
Oct 30 22:44:14.179953 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:44:14.179791    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0664869-236d-4245-8d16-374b217dda8d-kube-api-access-jpqzn\" (UniqueName: \"kubernetes.io/projected/c0664869-236d-4245-8d16-374b217dda8d-kube-api-access-jpqzn\") pod \"c0664869-236d-4245-8d16-374b217dda8d\" (UID: \"c0664869-236d-4245-8d16-374b217dda8d\") "
Oct 30 22:44:14.179953 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:44:14.179904    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0664869-236d-4245-8d16-374b217dda8d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0664869-236d-4245-8d16-374b217dda8d-maas-api-openshift-ca\") pod \"c0664869-236d-4245-8d16-374b217dda8d\" (UID: \"c0664869-236d-4245-8d16-374b217dda8d\") "
Oct 30 22:44:14.180277 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:44:14.180238    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0664869-236d-4245-8d16-374b217dda8d-maas-api-openshift-ca" pod "c0664869-236d-4245-8d16-374b217dda8d" (UID: "c0664869-236d-4245-8d16-374b217dda8d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:44:14.182034 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:44:14.182007    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0664869-236d-4245-8d16-374b217dda8d-kube-api-access-jpqzn" pod "c0664869-236d-4245-8d16-374b217dda8d" (UID: "c0664869-236d-4245-8d16-374b217dda8d"). InnerVolumeSpecName "kube-api-access-jpqzn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:44:14.182292 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:44:14.182263    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0664869-236d-4245-8d16-374b217dda8d-maas-api-xks-ca" pod "c0664869-236d-4245-8d16-374b217dda8d" (UID: "c0664869-236d-4245-8d16-374b217dda8d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:44:14.280918 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:44:14.280831    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0664869-236d-4245-8d16-374b217dda8d-maas-api-xks-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:44:14.280918 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:44:14.280867    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpqzn\" (UniqueName: \"kubernetes.io/projected/c0664869-236d-4245-8d16-374b217dda8d-kube-api-access-jpqzn\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:44:14.280918 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:44:14.280879    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0664869-236d-4245-8d16-374b217dda8d-maas-api-openshift-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:44:14.975425 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:44:14.974522    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-2r5j6" event={"ID":"c0664869-236d-4245-8d16-374b217dda8d","Type":"ContainerDied","Data":"0787647ef3d754ce08bf540a1f68e5cdd66cc02134356edc7b68298f1647ea53"}
Oct 30 22:44:14.975425 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:44:14.974590    5678 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0787647ef3d754ce08bf540a1f68e5cdd66cc02134356edc7b68298f1647ea53"
Oct 30 22:44:14.975425 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:44:14.974661    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-2r5j6"
Oct 30 22:44:40.571089 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:44:40.571012    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 22:44:45.927648 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:44:45.927611    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-n7chv"]
Oct 30 22:44:45.928181 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:44:45.928023    5678 cpu_manager.go:414] "containerMap: removing container" podUID="c0664869-236d-4245-8d16-374b217dda8d" containerName="revoke-keys"
Oct 30 22:44:45.928181 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:44:45.928035    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0664869-236d-4245-8d16-374b217dda8d" containerName="revoke-keys"
Oct 30 22:44:45.928181 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:44:45.928086    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0664869-236d-4245-8d16-374b217dda8d" containerName="revoke-keys"
Oct 30 22:44:45.931464 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:44:45.931443    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-n7chv"
Oct 30 22:44:45.983688 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:44:45.983659    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:44:46.030771 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:44:46.030742    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:44:46.030934 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:44:46.030784    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:44:46.030934 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:44:46.030871    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5rvcj\""
Oct 30 22:44:46.032984 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:44:46.032965    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-n7chv"]
Oct 30 22:44:46.108317 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:44:46.108289    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4afea247-e8e1-455e-ae19-6e9c4282ba84-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-n7chv\" (UID: \"4afea247-e8e1-455e-ae19-6e9c4282ba84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-n7chv"
Oct 30 22:44:46.108493 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:44:46.108324    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4afea247-e8e1-455e-ae19-6e9c4282ba84-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-n7chv\" (UID: \"4afea247-e8e1-455e-ae19-6e9c4282ba84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-n7chv"
Oct 30 22:44:46.108493 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:44:46.108348    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4tzs\" (UniqueName: \"kubernetes.io/projected/4afea247-e8e1-455e-ae19-6e9c4282ba84-kube-api-access-l4tzs\") pod \"maas-api-revoke-sub-ae594979-n7chv\" (UID: \"4afea247-e8e1-455e-ae19-6e9c4282ba84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-n7chv"
Oct 30 22:44:46.209907 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:44:46.209818    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4afea247-e8e1-455e-ae19-6e9c4282ba84-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-n7chv\" (UID: \"4afea247-e8e1-455e-ae19-6e9c4282ba84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-n7chv"
Oct 30 22:44:46.209907 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:44:46.209856    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4afea247-e8e1-455e-ae19-6e9c4282ba84-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-n7chv\" (UID: \"4afea247-e8e1-455e-ae19-6e9c4282ba84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-n7chv"
Oct 30 22:44:46.209907 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:44:46.209883    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4tzs\" (UniqueName: \"kubernetes.io/projected/4afea247-e8e1-455e-ae19-6e9c4282ba84-kube-api-access-l4tzs\") pod \"maas-api-revoke-sub-ae594979-n7chv\" (UID: \"4afea247-e8e1-455e-ae19-6e9c4282ba84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-n7chv"
Oct 30 22:44:46.210567 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:44:46.210544    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4afea247-e8e1-455e-ae19-6e9c4282ba84-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-n7chv\" (UID: \"4afea247-e8e1-455e-ae19-6e9c4282ba84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-n7chv"
Oct 30 22:44:46.212273 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:44:46.212253    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4afea247-e8e1-455e-ae19-6e9c4282ba84-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-n7chv\" (UID: \"4afea247-e8e1-455e-ae19-6e9c4282ba84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-n7chv"
Oct 30 22:44:46.268966 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:44:46.268936    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4tzs\" (UniqueName: \"kubernetes.io/projected/4afea247-e8e1-455e-ae19-6e9c4282ba84-kube-api-access-l4tzs\") pod \"maas-api-revoke-sub-ae594979-n7chv\" (UID: \"4afea247-e8e1-455e-ae19-6e9c4282ba84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-n7chv"
Oct 30 22:44:46.543838 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:44:46.543763    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-n7chv"
Oct 30 22:44:46.927349 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:44:46.927312    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-n7chv"]
Oct 30 22:44:46.934620 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:44:46.934590    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4afea247_e8e1_455e_ae19_6e9c4282ba84.slice/crio-e3ad3c43f29c377fd56f7de0480c3ba896273f49bb31a70c1b2434d95fd4774c WatchSource:0}: Error finding container e3ad3c43f29c377fd56f7de0480c3ba896273f49bb31a70c1b2434d95fd4774c: Status 404 returned error can't find the container with id e3ad3c43f29c377fd56f7de0480c3ba896273f49bb31a70c1b2434d95fd4774c
Oct 30 22:44:47.173701 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:44:47.173672    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-n7chv" event={"ID":"4afea247-e8e1-455e-ae19-6e9c4282ba84","Type":"ContainerStarted","Data":"e3ad3c43f29c377fd56f7de0480c3ba896273f49bb31a70c1b2434d95fd4774c"}
Oct 30 22:44:48.181053 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:44:48.181016    5678 generic.go:356] "Generic (PLEG): container finished" podID="4afea247-e8e1-455e-ae19-6e9c4282ba84" containerID="a25a9f52dcf77517d2278ec6cedf32bfe060cd14547d55cd90fad492a773b14c" exitCode=0
Oct 30 22:44:48.181523 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:44:48.181102    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-n7chv" event={"ID":"4afea247-e8e1-455e-ae19-6e9c4282ba84","Type":"ContainerDied","Data":"a25a9f52dcf77517d2278ec6cedf32bfe060cd14547d55cd90fad492a773b14c"}
Oct 30 22:44:49.313412 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:44:49.313374    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-n7chv"
Oct 30 22:44:49.331526 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:44:49.331504    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4afea247-e8e1-455e-ae19-6e9c4282ba84-kube-api-access-l4tzs\" (UniqueName: \"kubernetes.io/projected/4afea247-e8e1-455e-ae19-6e9c4282ba84-kube-api-access-l4tzs\") pod \"4afea247-e8e1-455e-ae19-6e9c4282ba84\" (UID: \"4afea247-e8e1-455e-ae19-6e9c4282ba84\") "
Oct 30 22:44:49.331592 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:44:49.331549    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4afea247-e8e1-455e-ae19-6e9c4282ba84-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4afea247-e8e1-455e-ae19-6e9c4282ba84-maas-api-openshift-ca\") pod \"4afea247-e8e1-455e-ae19-6e9c4282ba84\" (UID: \"4afea247-e8e1-455e-ae19-6e9c4282ba84\") "
Oct 30 22:44:49.331592 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:44:49.331578    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4afea247-e8e1-455e-ae19-6e9c4282ba84-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4afea247-e8e1-455e-ae19-6e9c4282ba84-maas-api-xks-ca\") pod \"4afea247-e8e1-455e-ae19-6e9c4282ba84\" (UID: \"4afea247-e8e1-455e-ae19-6e9c4282ba84\") "
Oct 30 22:44:49.331910 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:44:49.331886    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4afea247-e8e1-455e-ae19-6e9c4282ba84-maas-api-openshift-ca" pod "4afea247-e8e1-455e-ae19-6e9c4282ba84" (UID: "4afea247-e8e1-455e-ae19-6e9c4282ba84"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:44:49.333479 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:44:49.333458    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4afea247-e8e1-455e-ae19-6e9c4282ba84-kube-api-access-l4tzs" pod "4afea247-e8e1-455e-ae19-6e9c4282ba84" (UID: "4afea247-e8e1-455e-ae19-6e9c4282ba84"). InnerVolumeSpecName "kube-api-access-l4tzs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:44:49.333549 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:44:49.333464    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4afea247-e8e1-455e-ae19-6e9c4282ba84-maas-api-xks-ca" pod "4afea247-e8e1-455e-ae19-6e9c4282ba84" (UID: "4afea247-e8e1-455e-ae19-6e9c4282ba84"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:44:49.432558 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:44:49.432528    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4tzs\" (UniqueName: \"kubernetes.io/projected/4afea247-e8e1-455e-ae19-6e9c4282ba84-kube-api-access-l4tzs\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:44:49.432558 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:44:49.432557    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4afea247-e8e1-455e-ae19-6e9c4282ba84-maas-api-openshift-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:44:49.432558 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:44:49.432572    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4afea247-e8e1-455e-ae19-6e9c4282ba84-maas-api-xks-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:44:50.192784 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:44:50.192752    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-n7chv"
Oct 30 22:44:50.192784 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:44:50.192779    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-n7chv" event={"ID":"4afea247-e8e1-455e-ae19-6e9c4282ba84","Type":"ContainerDied","Data":"e3ad3c43f29c377fd56f7de0480c3ba896273f49bb31a70c1b2434d95fd4774c"}
Oct 30 22:44:50.192998 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:44:50.192826    5678 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e3ad3c43f29c377fd56f7de0480c3ba896273f49bb31a70c1b2434d95fd4774c"
Oct 30 22:45:00.175489 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:45:00.175454    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890005-jpwgp"]
Oct 30 22:45:00.175945 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:45:00.175910    5678 cpu_manager.go:414] "containerMap: removing container" podUID="4afea247-e8e1-455e-ae19-6e9c4282ba84" containerName="revoke-keys"
Oct 30 22:45:00.175945 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:45:00.175929    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4afea247-e8e1-455e-ae19-6e9c4282ba84" containerName="revoke-keys"
Oct 30 22:45:00.176041 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:45:00.176008    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="4afea247-e8e1-455e-ae19-6e9c4282ba84" containerName="revoke-keys"
Oct 30 22:45:00.179096 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:45:00.179078    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-jpwgp"
Oct 30 22:45:00.239095 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:45:00.239052    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8vm2\" (UniqueName: \"kubernetes.io/projected/40d0015d-699b-4aee-884e-68d11838d57b-kube-api-access-q8vm2\") pod \"maas-api-key-cleanup-29890005-jpwgp\" (UID: \"40d0015d-699b-4aee-884e-68d11838d57b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-jpwgp"
Oct 30 22:45:00.239296 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:45:00.239150    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40d0015d-699b-4aee-884e-68d11838d57b-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890005-jpwgp\" (UID: \"40d0015d-699b-4aee-884e-68d11838d57b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-jpwgp"
Oct 30 22:45:00.277516 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:45:00.277485    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5rvcj\""
Oct 30 22:45:00.277516 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:45:00.277488    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:45:00.277516 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:45:00.277518    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:45:00.279857 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:45:00.279828    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890005-jpwgp"]
Oct 30 22:45:00.339871 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:45:00.339835    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8vm2\" (UniqueName: \"kubernetes.io/projected/40d0015d-699b-4aee-884e-68d11838d57b-kube-api-access-q8vm2\") pod \"maas-api-key-cleanup-29890005-jpwgp\" (UID: \"40d0015d-699b-4aee-884e-68d11838d57b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-jpwgp"
Oct 30 22:45:00.340041 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:45:00.339893    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40d0015d-699b-4aee-884e-68d11838d57b-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890005-jpwgp\" (UID: \"40d0015d-699b-4aee-884e-68d11838d57b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-jpwgp"
Oct 30 22:45:00.340551 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:45:00.340534    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40d0015d-699b-4aee-884e-68d11838d57b-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890005-jpwgp\" (UID: \"40d0015d-699b-4aee-884e-68d11838d57b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-jpwgp"
Oct 30 22:45:00.398504 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:45:00.398469    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8vm2\" (UniqueName: \"kubernetes.io/projected/40d0015d-699b-4aee-884e-68d11838d57b-kube-api-access-q8vm2\") pod \"maas-api-key-cleanup-29890005-jpwgp\" (UID: \"40d0015d-699b-4aee-884e-68d11838d57b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-jpwgp"
Oct 30 22:45:00.496927 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:45:00.496853    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-jpwgp"
Oct 30 22:45:00.884579 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:45:00.884547    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890005-jpwgp"]
Oct 30 22:45:00.892733 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:45:00.892688    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40d0015d_699b_4aee_884e_68d11838d57b.slice/crio-4f1d855e3481aa82ba8b59bf8eca4a9846b31dfd12673702db6cfe51f0fda9ea WatchSource:0}: Error finding container 4f1d855e3481aa82ba8b59bf8eca4a9846b31dfd12673702db6cfe51f0fda9ea: Status 404 returned error can't find the container with id 4f1d855e3481aa82ba8b59bf8eca4a9846b31dfd12673702db6cfe51f0fda9ea
Oct 30 22:45:01.252037 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:45:01.252005    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-jpwgp" event={"ID":"40d0015d-699b-4aee-884e-68d11838d57b","Type":"ContainerStarted","Data":"0e501332f080697b678d63834c6941a5c0557d82e6712ab6dfdc4f4c81892ba9"}
Oct 30 22:45:01.252468 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:45:01.252062    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-jpwgp" event={"ID":"40d0015d-699b-4aee-884e-68d11838d57b","Type":"ContainerStarted","Data":"4f1d855e3481aa82ba8b59bf8eca4a9846b31dfd12673702db6cfe51f0fda9ea"}
Oct 30 22:45:01.363337 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:45:01.363281    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-jpwgp" podStartSLOduration=1.363267451 podStartE2EDuration="1.363267451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:45:01.362146742 +0000 UTC m=+3278.966454295" watchObservedRunningTime="2026-10-30 22:45:01.363267451 +0000 UTC m=+3278.967575054"
Oct 30 22:45:02.259118 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:45:02.259083    5678 generic.go:356] "Generic (PLEG): container finished" podID="40d0015d-699b-4aee-884e-68d11838d57b" containerID="0e501332f080697b678d63834c6941a5c0557d82e6712ab6dfdc4f4c81892ba9" exitCode=60
Oct 30 22:45:02.259648 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:45:02.259165    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-jpwgp" event={"ID":"40d0015d-699b-4aee-884e-68d11838d57b","Type":"ContainerDied","Data":"0e501332f080697b678d63834c6941a5c0557d82e6712ab6dfdc4f4c81892ba9"}
Oct 30 22:45:02.259648 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:45:02.259430    5678 scope.go:122] "RemoveContainer" containerID="0e501332f080697b678d63834c6941a5c0557d82e6712ab6dfdc4f4c81892ba9"
Oct 30 22:45:03.266896 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:45:03.266812    5678 generic.go:356] "Generic (PLEG): container finished" podID="40d0015d-699b-4aee-884e-68d11838d57b" containerID="5f3beccfe6c0f25780c60a59971153194e588f8fefc0a0a4226651fe26ecace5" exitCode=60
Oct 30 22:45:03.266896 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:45:03.266862    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-jpwgp" event={"ID":"40d0015d-699b-4aee-884e-68d11838d57b","Type":"ContainerDied","Data":"5f3beccfe6c0f25780c60a59971153194e588f8fefc0a0a4226651fe26ecace5"}
Oct 30 22:45:03.267375 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:45:03.266910    5678 scope.go:122] "RemoveContainer" containerID="0e501332f080697b678d63834c6941a5c0557d82e6712ab6dfdc4f4c81892ba9"
Oct 30 22:45:03.267375 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:45:03.267113    5678 scope.go:122] "RemoveContainer" containerID="5f3beccfe6c0f25780c60a59971153194e588f8fefc0a0a4226651fe26ecace5"
Oct 30 22:45:03.267375 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:45:03.267330    5678 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890005-jpwgp_odh-ai-gateway-infra(40d0015d-699b-4aee-884e-68d11838d57b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-jpwgp" podUID="40d0015d-699b-4aee-884e-68d11838d57b"
Oct 30 22:45:04.275983 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:45:04.275936    5678 scope.go:122] "RemoveContainer" containerID="5f3beccfe6c0f25780c60a59971153194e588f8fefc0a0a4226651fe26ecace5"
Oct 30 22:45:04.276476 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:45:04.276195    5678 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890005-jpwgp_odh-ai-gateway-infra(40d0015d-699b-4aee-884e-68d11838d57b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-jpwgp" podUID="40d0015d-699b-4aee-884e-68d11838d57b"
Oct 30 22:45:06.339647 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:45:06.339597    5678 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-8mdvr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:45:06.340189 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:45:06.339699    5678 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8mdvr" podUID="c4729d87-68c3-4c11-9a1d-c235bc370104" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:45:10.570850 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:45:10.570816    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 22:45:13.516365 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:45:13.516331    5678 scope.go:122] "RemoveContainer" containerID="5f3beccfe6c0f25780c60a59971153194e588f8fefc0a0a4226651fe26ecace5"
Oct 30 22:45:14.341464 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:45:14.341435    5678 generic.go:356] "Generic (PLEG): container finished" podID="40d0015d-699b-4aee-884e-68d11838d57b" containerID="7af5407bab060f88ec0dc77a5a9e95e5863b72953b6bf2a402629936fdb02546" exitCode=60
Oct 30 22:45:14.341590 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:45:14.341476    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-jpwgp" event={"ID":"40d0015d-699b-4aee-884e-68d11838d57b","Type":"ContainerDied","Data":"7af5407bab060f88ec0dc77a5a9e95e5863b72953b6bf2a402629936fdb02546"}
Oct 30 22:45:14.341590 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:45:14.341519    5678 scope.go:122] "RemoveContainer" containerID="5f3beccfe6c0f25780c60a59971153194e588f8fefc0a0a4226651fe26ecace5"
Oct 30 22:45:14.341872 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:45:14.341856    5678 scope.go:122] "RemoveContainer" containerID="7af5407bab060f88ec0dc77a5a9e95e5863b72953b6bf2a402629936fdb02546"
Oct 30 22:45:14.342153 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:45:14.342132    5678 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29890005-jpwgp_odh-ai-gateway-infra(40d0015d-699b-4aee-884e-68d11838d57b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-jpwgp" podUID="40d0015d-699b-4aee-884e-68d11838d57b"
Oct 30 22:45:15.467582 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:45:15.467551    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890005-jpwgp"]
Oct 30 22:45:15.602587 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:45:15.602564    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-jpwgp"
Oct 30 22:45:15.687052 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:45:15.687022    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40d0015d-699b-4aee-884e-68d11838d57b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40d0015d-699b-4aee-884e-68d11838d57b-maas-api-service-ca\") pod \"40d0015d-699b-4aee-884e-68d11838d57b\" (UID: \"40d0015d-699b-4aee-884e-68d11838d57b\") "
Oct 30 22:45:15.687235 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:45:15.687131    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40d0015d-699b-4aee-884e-68d11838d57b-kube-api-access-q8vm2\" (UniqueName: \"kubernetes.io/projected/40d0015d-699b-4aee-884e-68d11838d57b-kube-api-access-q8vm2\") pod \"40d0015d-699b-4aee-884e-68d11838d57b\" (UID: \"40d0015d-699b-4aee-884e-68d11838d57b\") "
Oct 30 22:45:15.687373 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:45:15.687349    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40d0015d-699b-4aee-884e-68d11838d57b-maas-api-service-ca" pod "40d0015d-699b-4aee-884e-68d11838d57b" (UID: "40d0015d-699b-4aee-884e-68d11838d57b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:45:15.687529 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:45:15.687513    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40d0015d-699b-4aee-884e-68d11838d57b-maas-api-service-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:45:15.689515 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:45:15.689493    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40d0015d-699b-4aee-884e-68d11838d57b-kube-api-access-q8vm2" pod "40d0015d-699b-4aee-884e-68d11838d57b" (UID: "40d0015d-699b-4aee-884e-68d11838d57b"). InnerVolumeSpecName "kube-api-access-q8vm2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:45:15.789018 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:45:15.788938    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8vm2\" (UniqueName: \"kubernetes.io/projected/40d0015d-699b-4aee-884e-68d11838d57b-kube-api-access-q8vm2\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:45:16.354804 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:45:16.354771    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-jpwgp"
Oct 30 22:45:16.354993 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:45:16.354800    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-jpwgp" event={"ID":"40d0015d-699b-4aee-884e-68d11838d57b","Type":"ContainerDied","Data":"4f1d855e3481aa82ba8b59bf8eca4a9846b31dfd12673702db6cfe51f0fda9ea"}
Oct 30 22:45:16.354993 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:45:16.354865    5678 scope.go:122] "RemoveContainer" containerID="7af5407bab060f88ec0dc77a5a9e95e5863b72953b6bf2a402629936fdb02546"
Oct 30 22:45:16.512269 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:45:16.512238    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890005-jpwgp"]
Oct 30 22:45:16.560147 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:45:16.560119    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890005-jpwgp"]
Oct 30 22:45:18.522203 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:45:18.522162    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40d0015d-699b-4aee-884e-68d11838d57b" path="/var/lib/kubelet/pods/40d0015d-699b-4aee-884e-68d11838d57b/volumes"
Oct 30 22:45:36.065494 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:45:36.065450    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28e87096-54mrw"]
Oct 30 22:45:36.108375 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:45:36.108339    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28e87096-54mrw"]
Oct 30 22:45:36.530111 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:45:36.530077    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e99e89f9-7b36-4876-883d-11432b7aa108" path="/var/lib/kubelet/pods/e99e89f9-7b36-4876-883d-11432b7aa108/volumes"
Oct 30 22:45:40.571100 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:45:40.571066    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 22:45:44.925559 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:45:44.925526    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-hv6cz"]
Oct 30 22:45:44.925933 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:45:44.925895    5678 cpu_manager.go:414] "containerMap: removing container" podUID="40d0015d-699b-4aee-884e-68d11838d57b" containerName="cleanup"
Oct 30 22:45:44.925933 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:45:44.925906    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40d0015d-699b-4aee-884e-68d11838d57b" containerName="cleanup"
Oct 30 22:45:44.925933 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:45:44.925916    5678 cpu_manager.go:414] "containerMap: removing container" podUID="40d0015d-699b-4aee-884e-68d11838d57b" containerName="cleanup"
Oct 30 22:45:44.925933 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:45:44.925920    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40d0015d-699b-4aee-884e-68d11838d57b" containerName="cleanup"
Oct 30 22:45:44.926072 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:45:44.926002    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="40d0015d-699b-4aee-884e-68d11838d57b" containerName="cleanup"
Oct 30 22:45:44.926072 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:45:44.926017    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="40d0015d-699b-4aee-884e-68d11838d57b" containerName="cleanup"
Oct 30 22:45:44.926072 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:45:44.926025    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="40d0015d-699b-4aee-884e-68d11838d57b" containerName="cleanup"
Oct 30 22:45:44.930588 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:45:44.930567    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-hv6cz"
Oct 30 22:45:44.984484 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:45:44.984459    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-hv6cz"]
Oct 30 22:45:45.028711 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:45:45.028682    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:45:45.028711 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:45:45.028692    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:45:45.028711 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:45:45.028706    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:45:45.028994 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:45:45.028747    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5rvcj\""
Oct 30 22:45:45.049469 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:45:45.049441    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14ada201-cmcmd"]
Oct 30 22:45:45.077775 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:45:45.077748    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14ada201-cmcmd"]
Oct 30 22:45:45.083101 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:45:45.083044    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pc9q\" (UniqueName: \"kubernetes.io/projected/62a9a748-fd94-4177-be46-c2cd40a08b98-kube-api-access-4pc9q\") pod \"maas-api-revoke-sub-79805ca9-hv6cz\" (UID: \"62a9a748-fd94-4177-be46-c2cd40a08b98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-hv6cz"
Oct 30 22:45:45.083101 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:45:45.083089    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62a9a748-fd94-4177-be46-c2cd40a08b98-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-hv6cz\" (UID: \"62a9a748-fd94-4177-be46-c2cd40a08b98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-hv6cz"
Oct 30 22:45:45.083263 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:45:45.083190    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62a9a748-fd94-4177-be46-c2cd40a08b98-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-hv6cz\" (UID: \"62a9a748-fd94-4177-be46-c2cd40a08b98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-hv6cz"
Oct 30 22:45:45.184323 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:45:45.184298    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pc9q\" (UniqueName: \"kubernetes.io/projected/62a9a748-fd94-4177-be46-c2cd40a08b98-kube-api-access-4pc9q\") pod \"maas-api-revoke-sub-79805ca9-hv6cz\" (UID: \"62a9a748-fd94-4177-be46-c2cd40a08b98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-hv6cz"
Oct 30 22:45:45.184490 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:45:45.184336    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62a9a748-fd94-4177-be46-c2cd40a08b98-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-hv6cz\" (UID: \"62a9a748-fd94-4177-be46-c2cd40a08b98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-hv6cz"
Oct 30 22:45:45.184490 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:45:45.184386    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62a9a748-fd94-4177-be46-c2cd40a08b98-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-hv6cz\" (UID: \"62a9a748-fd94-4177-be46-c2cd40a08b98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-hv6cz"
Oct 30 22:45:45.185046 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:45:45.185023    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62a9a748-fd94-4177-be46-c2cd40a08b98-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-hv6cz\" (UID: \"62a9a748-fd94-4177-be46-c2cd40a08b98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-hv6cz"
Oct 30 22:45:45.186846 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:45:45.186824    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62a9a748-fd94-4177-be46-c2cd40a08b98-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-hv6cz\" (UID: \"62a9a748-fd94-4177-be46-c2cd40a08b98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-hv6cz"
Oct 30 22:45:45.242819 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:45:45.242794    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pc9q\" (UniqueName: \"kubernetes.io/projected/62a9a748-fd94-4177-be46-c2cd40a08b98-kube-api-access-4pc9q\") pod \"maas-api-revoke-sub-79805ca9-hv6cz\" (UID: \"62a9a748-fd94-4177-be46-c2cd40a08b98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-hv6cz"
Oct 30 22:45:45.242938 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:45:45.242925    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-hv6cz"
Oct 30 22:45:45.623663 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:45:45.623630    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-hv6cz"]
Oct 30 22:45:45.630910 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:45:45.630883    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62a9a748_fd94_4177_be46_c2cd40a08b98.slice/crio-87f524f16c04c0f68f83fd47538911b3460a561f191a283329d41875d064e180 WatchSource:0}: Error finding container 87f524f16c04c0f68f83fd47538911b3460a561f191a283329d41875d064e180: Status 404 returned error can't find the container with id 87f524f16c04c0f68f83fd47538911b3460a561f191a283329d41875d064e180
Oct 30 22:45:46.522103 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:45:46.522072    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63984586-39b2-4924-96eb-f0890de5ed03" path="/var/lib/kubelet/pods/63984586-39b2-4924-96eb-f0890de5ed03/volumes"
Oct 30 22:45:46.540711 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:45:46.540684    5678 generic.go:356] "Generic (PLEG): container finished" podID="62a9a748-fd94-4177-be46-c2cd40a08b98" containerID="13f8b4a28e7777f84945526541938e6b660e6a8ca3f42f94beb4a7551ebe5a32" exitCode=0
Oct 30 22:45:46.540839 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:45:46.540760    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-hv6cz" event={"ID":"62a9a748-fd94-4177-be46-c2cd40a08b98","Type":"ContainerDied","Data":"13f8b4a28e7777f84945526541938e6b660e6a8ca3f42f94beb4a7551ebe5a32"}
Oct 30 22:45:46.540839 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:45:46.540806    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-hv6cz" event={"ID":"62a9a748-fd94-4177-be46-c2cd40a08b98","Type":"ContainerStarted","Data":"87f524f16c04c0f68f83fd47538911b3460a561f191a283329d41875d064e180"}
Oct 30 22:45:47.686650 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:45:47.686622    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-hv6cz"
Oct 30 22:45:47.707647 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:45:47.707621    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62a9a748-fd94-4177-be46-c2cd40a08b98-kube-api-access-4pc9q\" (UniqueName: \"kubernetes.io/projected/62a9a748-fd94-4177-be46-c2cd40a08b98-kube-api-access-4pc9q\") pod \"62a9a748-fd94-4177-be46-c2cd40a08b98\" (UID: \"62a9a748-fd94-4177-be46-c2cd40a08b98\") "
Oct 30 22:45:47.707783 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:45:47.707681    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/62a9a748-fd94-4177-be46-c2cd40a08b98-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62a9a748-fd94-4177-be46-c2cd40a08b98-maas-api-xks-ca\") pod \"62a9a748-fd94-4177-be46-c2cd40a08b98\" (UID: \"62a9a748-fd94-4177-be46-c2cd40a08b98\") "
Oct 30 22:45:47.707783 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:45:47.707744    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62a9a748-fd94-4177-be46-c2cd40a08b98-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62a9a748-fd94-4177-be46-c2cd40a08b98-maas-api-openshift-ca\") pod \"62a9a748-fd94-4177-be46-c2cd40a08b98\" (UID: \"62a9a748-fd94-4177-be46-c2cd40a08b98\") "
Oct 30 22:45:47.708188 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:45:47.708160    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62a9a748-fd94-4177-be46-c2cd40a08b98-maas-api-openshift-ca" pod "62a9a748-fd94-4177-be46-c2cd40a08b98" (UID: "62a9a748-fd94-4177-be46-c2cd40a08b98"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:45:47.709675 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:45:47.709651    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62a9a748-fd94-4177-be46-c2cd40a08b98-maas-api-xks-ca" pod "62a9a748-fd94-4177-be46-c2cd40a08b98" (UID: "62a9a748-fd94-4177-be46-c2cd40a08b98"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:45:47.709780 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:45:47.709719    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62a9a748-fd94-4177-be46-c2cd40a08b98-kube-api-access-4pc9q" pod "62a9a748-fd94-4177-be46-c2cd40a08b98" (UID: "62a9a748-fd94-4177-be46-c2cd40a08b98"). InnerVolumeSpecName "kube-api-access-4pc9q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:45:47.808788 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:45:47.808711    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62a9a748-fd94-4177-be46-c2cd40a08b98-maas-api-openshift-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:45:47.808788 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:45:47.808752    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pc9q\" (UniqueName: \"kubernetes.io/projected/62a9a748-fd94-4177-be46-c2cd40a08b98-kube-api-access-4pc9q\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:45:47.808788 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:45:47.808768    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62a9a748-fd94-4177-be46-c2cd40a08b98-maas-api-xks-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:45:48.552673 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:45:48.552646    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-hv6cz"
Oct 30 22:45:48.552824 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:45:48.552670    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-hv6cz" event={"ID":"62a9a748-fd94-4177-be46-c2cd40a08b98","Type":"ContainerDied","Data":"87f524f16c04c0f68f83fd47538911b3460a561f191a283329d41875d064e180"}
Oct 30 22:45:48.552824 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:45:48.552714    5678 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="87f524f16c04c0f68f83fd47538911b3460a561f191a283329d41875d064e180"
Oct 30 22:46:08.726676 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:46:08.726586    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-8zfl6"]
Oct 30 22:46:08.727278 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:46:08.727006    5678 cpu_manager.go:414] "containerMap: removing container" podUID="40d0015d-699b-4aee-884e-68d11838d57b" containerName="cleanup"
Oct 30 22:46:08.727278 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:46:08.727021    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40d0015d-699b-4aee-884e-68d11838d57b" containerName="cleanup"
Oct 30 22:46:08.727278 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:46:08.727045    5678 cpu_manager.go:414] "containerMap: removing container" podUID="62a9a748-fd94-4177-be46-c2cd40a08b98" containerName="revoke-keys"
Oct 30 22:46:08.727278 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:46:08.727051    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62a9a748-fd94-4177-be46-c2cd40a08b98" containerName="revoke-keys"
Oct 30 22:46:08.727278 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:46:08.727127    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="62a9a748-fd94-4177-be46-c2cd40a08b98" containerName="revoke-keys"
Oct 30 22:46:08.730300 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:46:08.730247    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-8zfl6"
Oct 30 22:46:08.786179 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:46:08.786149    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-8zfl6"]
Oct 30 22:46:08.823139 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:46:08.823113    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhfxq\" (UniqueName: \"kubernetes.io/projected/1c6e09de-bf26-4a43-a8fc-5a2bdf2ec10c-kube-api-access-zhfxq\") pod \"maas-api-revoke-sub-0b7e8edb-8zfl6\" (UID: \"1c6e09de-bf26-4a43-a8fc-5a2bdf2ec10c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-8zfl6"
Oct 30 22:46:08.823298 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:46:08.823149    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c6e09de-bf26-4a43-a8fc-5a2bdf2ec10c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-8zfl6\" (UID: \"1c6e09de-bf26-4a43-a8fc-5a2bdf2ec10c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-8zfl6"
Oct 30 22:46:08.823298 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:46:08.823174    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c6e09de-bf26-4a43-a8fc-5a2bdf2ec10c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-8zfl6\" (UID: \"1c6e09de-bf26-4a43-a8fc-5a2bdf2ec10c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-8zfl6"
Oct 30 22:46:08.829900 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:46:08.829860    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5rvcj\""
Oct 30 22:46:08.829900 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:46:08.829871    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:46:08.829900 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:46:08.829884    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:46:08.829900 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:46:08.829892    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:46:08.924480 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:46:08.924452    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhfxq\" (UniqueName: \"kubernetes.io/projected/1c6e09de-bf26-4a43-a8fc-5a2bdf2ec10c-kube-api-access-zhfxq\") pod \"maas-api-revoke-sub-0b7e8edb-8zfl6\" (UID: \"1c6e09de-bf26-4a43-a8fc-5a2bdf2ec10c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-8zfl6"
Oct 30 22:46:08.924480 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:46:08.924487    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c6e09de-bf26-4a43-a8fc-5a2bdf2ec10c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-8zfl6\" (UID: \"1c6e09de-bf26-4a43-a8fc-5a2bdf2ec10c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-8zfl6"
Oct 30 22:46:08.924696 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:46:08.924509    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c6e09de-bf26-4a43-a8fc-5a2bdf2ec10c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-8zfl6\" (UID: \"1c6e09de-bf26-4a43-a8fc-5a2bdf2ec10c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-8zfl6"
Oct 30 22:46:08.925177 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:46:08.925158    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c6e09de-bf26-4a43-a8fc-5a2bdf2ec10c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-8zfl6\" (UID: \"1c6e09de-bf26-4a43-a8fc-5a2bdf2ec10c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-8zfl6"
Oct 30 22:46:08.926764 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:46:08.926746    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c6e09de-bf26-4a43-a8fc-5a2bdf2ec10c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-8zfl6\" (UID: \"1c6e09de-bf26-4a43-a8fc-5a2bdf2ec10c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-8zfl6"
Oct 30 22:46:08.984071 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:46:08.983999    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhfxq\" (UniqueName: \"kubernetes.io/projected/1c6e09de-bf26-4a43-a8fc-5a2bdf2ec10c-kube-api-access-zhfxq\") pod \"maas-api-revoke-sub-0b7e8edb-8zfl6\" (UID: \"1c6e09de-bf26-4a43-a8fc-5a2bdf2ec10c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-8zfl6"
Oct 30 22:46:09.043206 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:46:09.043173    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-8zfl6"
Oct 30 22:46:09.429785 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:46:09.425003    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-8zfl6"]
Oct 30 22:46:09.435791 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:46:09.435761    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c6e09de_bf26_4a43_a8fc_5a2bdf2ec10c.slice/crio-8c7707b1724c2f8f9891a326b5a5217ad006da3be338c055c2022e8681e70f6e WatchSource:0}: Error finding container 8c7707b1724c2f8f9891a326b5a5217ad006da3be338c055c2022e8681e70f6e: Status 404 returned error can't find the container with id 8c7707b1724c2f8f9891a326b5a5217ad006da3be338c055c2022e8681e70f6e
Oct 30 22:46:09.661882 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:46:09.661847    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-8zfl6" event={"ID":"1c6e09de-bf26-4a43-a8fc-5a2bdf2ec10c","Type":"ContainerStarted","Data":"8c7707b1724c2f8f9891a326b5a5217ad006da3be338c055c2022e8681e70f6e"}
Oct 30 22:46:10.571018 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:46:10.570939    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 22:46:10.669603 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:46:10.669569    5678 generic.go:356] "Generic (PLEG): container finished" podID="1c6e09de-bf26-4a43-a8fc-5a2bdf2ec10c" containerID="6fca4c5799b297bee1627f65e3808b5ee801425223cc7f33a1482b5b984827b0" exitCode=0
Oct 30 22:46:10.669761 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:46:10.669638    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-8zfl6" event={"ID":"1c6e09de-bf26-4a43-a8fc-5a2bdf2ec10c","Type":"ContainerDied","Data":"6fca4c5799b297bee1627f65e3808b5ee801425223cc7f33a1482b5b984827b0"}
Oct 30 22:46:11.802778 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:46:11.802753    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-8zfl6"
Oct 30 22:46:11.953697 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:46:11.953606    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c6e09de-bf26-4a43-a8fc-5a2bdf2ec10c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c6e09de-bf26-4a43-a8fc-5a2bdf2ec10c-maas-api-xks-ca\") pod \"1c6e09de-bf26-4a43-a8fc-5a2bdf2ec10c\" (UID: \"1c6e09de-bf26-4a43-a8fc-5a2bdf2ec10c\") "
Oct 30 22:46:11.953697 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:46:11.953668    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c6e09de-bf26-4a43-a8fc-5a2bdf2ec10c-kube-api-access-zhfxq\" (UniqueName: \"kubernetes.io/projected/1c6e09de-bf26-4a43-a8fc-5a2bdf2ec10c-kube-api-access-zhfxq\") pod \"1c6e09de-bf26-4a43-a8fc-5a2bdf2ec10c\" (UID: \"1c6e09de-bf26-4a43-a8fc-5a2bdf2ec10c\") "
Oct 30 22:46:11.953939 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:46:11.953807    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c6e09de-bf26-4a43-a8fc-5a2bdf2ec10c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c6e09de-bf26-4a43-a8fc-5a2bdf2ec10c-maas-api-openshift-ca\") pod \"1c6e09de-bf26-4a43-a8fc-5a2bdf2ec10c\" (UID: \"1c6e09de-bf26-4a43-a8fc-5a2bdf2ec10c\") "
Oct 30 22:46:11.954170 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:46:11.954145    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c6e09de-bf26-4a43-a8fc-5a2bdf2ec10c-maas-api-openshift-ca" pod "1c6e09de-bf26-4a43-a8fc-5a2bdf2ec10c" (UID: "1c6e09de-bf26-4a43-a8fc-5a2bdf2ec10c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:46:11.955757 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:46:11.955724    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c6e09de-bf26-4a43-a8fc-5a2bdf2ec10c-maas-api-xks-ca" pod "1c6e09de-bf26-4a43-a8fc-5a2bdf2ec10c" (UID: "1c6e09de-bf26-4a43-a8fc-5a2bdf2ec10c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:46:11.955873 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:46:11.955777    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c6e09de-bf26-4a43-a8fc-5a2bdf2ec10c-kube-api-access-zhfxq" pod "1c6e09de-bf26-4a43-a8fc-5a2bdf2ec10c" (UID: "1c6e09de-bf26-4a43-a8fc-5a2bdf2ec10c"). InnerVolumeSpecName "kube-api-access-zhfxq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:46:12.055284 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:46:12.055243    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c6e09de-bf26-4a43-a8fc-5a2bdf2ec10c-maas-api-xks-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:46:12.055284 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:46:12.055274    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhfxq\" (UniqueName: \"kubernetes.io/projected/1c6e09de-bf26-4a43-a8fc-5a2bdf2ec10c-kube-api-access-zhfxq\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:46:12.055284 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:46:12.055286    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c6e09de-bf26-4a43-a8fc-5a2bdf2ec10c-maas-api-openshift-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:46:12.680510 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:46:12.680474    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-8zfl6" event={"ID":"1c6e09de-bf26-4a43-a8fc-5a2bdf2ec10c","Type":"ContainerDied","Data":"8c7707b1724c2f8f9891a326b5a5217ad006da3be338c055c2022e8681e70f6e"}
Oct 30 22:46:12.680510 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:46:12.680499    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-8zfl6"
Oct 30 22:46:12.680723 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:46:12.680530    5678 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8c7707b1724c2f8f9891a326b5a5217ad006da3be338c055c2022e8681e70f6e"
Oct 30 22:46:20.073560 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:46:20.073520    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v7q4p"]
Oct 30 22:46:20.074134 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:46:20.074113    5678 cpu_manager.go:414] "containerMap: removing container" podUID="1c6e09de-bf26-4a43-a8fc-5a2bdf2ec10c" containerName="revoke-keys"
Oct 30 22:46:20.074229 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:46:20.074137    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c6e09de-bf26-4a43-a8fc-5a2bdf2ec10c" containerName="revoke-keys"
Oct 30 22:46:20.074277 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:46:20.074265    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c6e09de-bf26-4a43-a8fc-5a2bdf2ec10c" containerName="revoke-keys"
Oct 30 22:46:20.077976 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:46:20.077952    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v7q4p"
Oct 30 22:46:20.128742 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:46:20.128699    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x947l\" (UniqueName: \"kubernetes.io/projected/f7470c91-4aae-43b0-a57f-a413090e2321-kube-api-access-x947l\") pod \"maas-api-revoke-sub-3d4156f1-v7q4p\" (UID: \"f7470c91-4aae-43b0-a57f-a413090e2321\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v7q4p"
Oct 30 22:46:20.128896 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:46:20.128809    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7470c91-4aae-43b0-a57f-a413090e2321-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-v7q4p\" (UID: \"f7470c91-4aae-43b0-a57f-a413090e2321\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v7q4p"
Oct 30 22:46:20.128896 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:46:20.128861    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7470c91-4aae-43b0-a57f-a413090e2321-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-v7q4p\" (UID: \"f7470c91-4aae-43b0-a57f-a413090e2321\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v7q4p"
Oct 30 22:46:20.178134 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:46:20.178098    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:46:20.178134 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:46:20.178099    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:46:20.178382 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:46:20.178131    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:46:20.178382 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:46:20.178156    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5rvcj\""
Oct 30 22:46:20.180549 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:46:20.180524    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v7q4p"]
Oct 30 22:46:20.230196 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:46:20.230169    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7470c91-4aae-43b0-a57f-a413090e2321-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-v7q4p\" (UID: \"f7470c91-4aae-43b0-a57f-a413090e2321\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v7q4p"
Oct 30 22:46:20.230344 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:46:20.230213    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7470c91-4aae-43b0-a57f-a413090e2321-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-v7q4p\" (UID: \"f7470c91-4aae-43b0-a57f-a413090e2321\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v7q4p"
Oct 30 22:46:20.230344 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:46:20.230262    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x947l\" (UniqueName: \"kubernetes.io/projected/f7470c91-4aae-43b0-a57f-a413090e2321-kube-api-access-x947l\") pod \"maas-api-revoke-sub-3d4156f1-v7q4p\" (UID: \"f7470c91-4aae-43b0-a57f-a413090e2321\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v7q4p"
Oct 30 22:46:20.230864 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:46:20.230837    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7470c91-4aae-43b0-a57f-a413090e2321-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-v7q4p\" (UID: \"f7470c91-4aae-43b0-a57f-a413090e2321\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v7q4p"
Oct 30 22:46:20.232629 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:46:20.232602    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7470c91-4aae-43b0-a57f-a413090e2321-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-v7q4p\" (UID: \"f7470c91-4aae-43b0-a57f-a413090e2321\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v7q4p"
Oct 30 22:46:20.291954 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:46:20.291920    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x947l\" (UniqueName: \"kubernetes.io/projected/f7470c91-4aae-43b0-a57f-a413090e2321-kube-api-access-x947l\") pod \"maas-api-revoke-sub-3d4156f1-v7q4p\" (UID: \"f7470c91-4aae-43b0-a57f-a413090e2321\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v7q4p"
Oct 30 22:46:20.390881 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:46:20.390855    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v7q4p"
Oct 30 22:46:20.777625 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:46:20.777545    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v7q4p"]
Oct 30 22:46:20.784909 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:46:20.784885    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7470c91_4aae_43b0_a57f_a413090e2321.slice/crio-dcb3ebc200a18a5706b2a8bc73d7eb22c58d70a7711dfbaa9fd855102895f6c2 WatchSource:0}: Error finding container dcb3ebc200a18a5706b2a8bc73d7eb22c58d70a7711dfbaa9fd855102895f6c2: Status 404 returned error can't find the container with id dcb3ebc200a18a5706b2a8bc73d7eb22c58d70a7711dfbaa9fd855102895f6c2
Oct 30 22:46:21.729955 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:46:21.729919    5678 generic.go:356] "Generic (PLEG): container finished" podID="f7470c91-4aae-43b0-a57f-a413090e2321" containerID="61a059d9b6b7117c4927686b0a2614357debe5e372ab66a6d9ec3eb7a63199d1" exitCode=0
Oct 30 22:46:21.730341 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:46:21.729960    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v7q4p" event={"ID":"f7470c91-4aae-43b0-a57f-a413090e2321","Type":"ContainerDied","Data":"61a059d9b6b7117c4927686b0a2614357debe5e372ab66a6d9ec3eb7a63199d1"}
Oct 30 22:46:21.730341 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:46:21.730008    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v7q4p" event={"ID":"f7470c91-4aae-43b0-a57f-a413090e2321","Type":"ContainerStarted","Data":"dcb3ebc200a18a5706b2a8bc73d7eb22c58d70a7711dfbaa9fd855102895f6c2"}
Oct 30 22:46:23.107612 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:46:23.107585    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v7q4p"
Oct 30 22:46:23.203883 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:46:23.203850    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7470c91-4aae-43b0-a57f-a413090e2321-kube-api-access-x947l\" (UniqueName: \"kubernetes.io/projected/f7470c91-4aae-43b0-a57f-a413090e2321-kube-api-access-x947l\") pod \"f7470c91-4aae-43b0-a57f-a413090e2321\" (UID: \"f7470c91-4aae-43b0-a57f-a413090e2321\") "
Oct 30 22:46:23.204091 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:46:23.203901    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7470c91-4aae-43b0-a57f-a413090e2321-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7470c91-4aae-43b0-a57f-a413090e2321-maas-api-xks-ca\") pod \"f7470c91-4aae-43b0-a57f-a413090e2321\" (UID: \"f7470c91-4aae-43b0-a57f-a413090e2321\") "
Oct 30 22:46:23.204091 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:46:23.204035    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7470c91-4aae-43b0-a57f-a413090e2321-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7470c91-4aae-43b0-a57f-a413090e2321-maas-api-openshift-ca\") pod \"f7470c91-4aae-43b0-a57f-a413090e2321\" (UID: \"f7470c91-4aae-43b0-a57f-a413090e2321\") "
Oct 30 22:46:23.204652 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:46:23.204622    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7470c91-4aae-43b0-a57f-a413090e2321-maas-api-openshift-ca" pod "f7470c91-4aae-43b0-a57f-a413090e2321" (UID: "f7470c91-4aae-43b0-a57f-a413090e2321"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:46:23.206289 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:46:23.206266    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7470c91-4aae-43b0-a57f-a413090e2321-maas-api-xks-ca" pod "f7470c91-4aae-43b0-a57f-a413090e2321" (UID: "f7470c91-4aae-43b0-a57f-a413090e2321"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:46:23.206393 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:46:23.206306    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7470c91-4aae-43b0-a57f-a413090e2321-kube-api-access-x947l" pod "f7470c91-4aae-43b0-a57f-a413090e2321" (UID: "f7470c91-4aae-43b0-a57f-a413090e2321"). InnerVolumeSpecName "kube-api-access-x947l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:46:23.305270 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:46:23.305229    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7470c91-4aae-43b0-a57f-a413090e2321-maas-api-openshift-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:46:23.305536 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:46:23.305516    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x947l\" (UniqueName: \"kubernetes.io/projected/f7470c91-4aae-43b0-a57f-a413090e2321-kube-api-access-x947l\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:46:23.305661 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:46:23.305646    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7470c91-4aae-43b0-a57f-a413090e2321-maas-api-xks-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:46:23.773433 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:46:23.771583    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v7q4p" event={"ID":"f7470c91-4aae-43b0-a57f-a413090e2321","Type":"ContainerDied","Data":"dcb3ebc200a18a5706b2a8bc73d7eb22c58d70a7711dfbaa9fd855102895f6c2"}
Oct 30 22:46:23.773433 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:46:23.771647    5678 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dcb3ebc200a18a5706b2a8bc73d7eb22c58d70a7711dfbaa9fd855102895f6c2"
Oct 30 22:46:23.773433 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:46:23.771720    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v7q4p"
Oct 30 22:46:36.074424 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:46:36.065698    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-wvmfj"]
Oct 30 22:46:36.101423 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:46:36.100363    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-wvmfj"]
Oct 30 22:46:36.525425 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:46:36.521449    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ac7a623-ad5e-44ab-8db5-5e9a69112e16" path="/var/lib/kubelet/pods/5ac7a623-ad5e-44ab-8db5-5e9a69112e16/volumes"
Oct 30 22:46:40.570774 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:46:40.570739    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 22:46:50.675666 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:46:50.675629    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v7q4p"]
Oct 30 22:46:50.722827 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:46:50.722796    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v7q4p"]
Oct 30 22:46:50.801141 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:46:50.801111    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-smnlr"]
Oct 30 22:46:50.801654 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:46:50.801639    5678 cpu_manager.go:414] "containerMap: removing container" podUID="f7470c91-4aae-43b0-a57f-a413090e2321" containerName="revoke-keys"
Oct 30 22:46:50.801709 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:46:50.801656    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7470c91-4aae-43b0-a57f-a413090e2321" containerName="revoke-keys"
Oct 30 22:46:50.801749 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:46:50.801744    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7470c91-4aae-43b0-a57f-a413090e2321" containerName="revoke-keys"
Oct 30 22:46:50.805928 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:46:50.805909    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-smnlr"
Oct 30 22:46:50.857688 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:46:50.857666    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:46:50.857688 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:46:50.857679    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:46:50.857688 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:46:50.857675    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:46:50.857948 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:46:50.857863    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5rvcj\""
Oct 30 22:46:50.860738 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:46:50.860714    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-smnlr"]
Oct 30 22:46:50.897483 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:46:50.897458    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzxd4\" (UniqueName: \"kubernetes.io/projected/c358407d-6272-41ed-8c9a-7a9705d51872-kube-api-access-wzxd4\") pod \"maas-api-revoke-sub-3d4156f1-smnlr\" (UID: \"c358407d-6272-41ed-8c9a-7a9705d51872\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-smnlr"
Oct 30 22:46:50.897595 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:46:50.897533    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c358407d-6272-41ed-8c9a-7a9705d51872-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-smnlr\" (UID: \"c358407d-6272-41ed-8c9a-7a9705d51872\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-smnlr"
Oct 30 22:46:50.897651 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:46:50.897632    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c358407d-6272-41ed-8c9a-7a9705d51872-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-smnlr\" (UID: \"c358407d-6272-41ed-8c9a-7a9705d51872\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-smnlr"
Oct 30 22:46:50.998355 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:46:50.998281    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c358407d-6272-41ed-8c9a-7a9705d51872-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-smnlr\" (UID: \"c358407d-6272-41ed-8c9a-7a9705d51872\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-smnlr"
Oct 30 22:46:50.998355 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:46:50.998335    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c358407d-6272-41ed-8c9a-7a9705d51872-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-smnlr\" (UID: \"c358407d-6272-41ed-8c9a-7a9705d51872\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-smnlr"
Oct 30 22:46:50.998543 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:46:50.998390    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzxd4\" (UniqueName: \"kubernetes.io/projected/c358407d-6272-41ed-8c9a-7a9705d51872-kube-api-access-wzxd4\") pod \"maas-api-revoke-sub-3d4156f1-smnlr\" (UID: \"c358407d-6272-41ed-8c9a-7a9705d51872\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-smnlr"
Oct 30 22:46:50.999010 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:46:50.998980    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c358407d-6272-41ed-8c9a-7a9705d51872-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-smnlr\" (UID: \"c358407d-6272-41ed-8c9a-7a9705d51872\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-smnlr"
Oct 30 22:46:51.000757 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:46:51.000732    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c358407d-6272-41ed-8c9a-7a9705d51872-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-smnlr\" (UID: \"c358407d-6272-41ed-8c9a-7a9705d51872\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-smnlr"
Oct 30 22:46:51.059820 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:46:51.059788    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzxd4\" (UniqueName: \"kubernetes.io/projected/c358407d-6272-41ed-8c9a-7a9705d51872-kube-api-access-wzxd4\") pod \"maas-api-revoke-sub-3d4156f1-smnlr\" (UID: \"c358407d-6272-41ed-8c9a-7a9705d51872\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-smnlr"
Oct 30 22:46:51.118670 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:46:51.118644    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-smnlr"
Oct 30 22:46:51.499146 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:46:51.499109    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-smnlr"]
Oct 30 22:46:51.506583 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:46:51.506554    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc358407d_6272_41ed_8c9a_7a9705d51872.slice/crio-cfd447c60c251c5c775f5258b93b45f1d8c386b057ad550d2edbd6186f8e9110 WatchSource:0}: Error finding container cfd447c60c251c5c775f5258b93b45f1d8c386b057ad550d2edbd6186f8e9110: Status 404 returned error can't find the container with id cfd447c60c251c5c775f5258b93b45f1d8c386b057ad550d2edbd6186f8e9110
Oct 30 22:46:51.946842 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:46:51.946808    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-smnlr" event={"ID":"c358407d-6272-41ed-8c9a-7a9705d51872","Type":"ContainerStarted","Data":"118df99c26939f10d7f38d8a632d850632d915b5056574c3bcd24ee55becc8b9"}
Oct 30 22:46:51.947337 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:46:51.946882    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-smnlr" event={"ID":"c358407d-6272-41ed-8c9a-7a9705d51872","Type":"ContainerStarted","Data":"cfd447c60c251c5c775f5258b93b45f1d8c386b057ad550d2edbd6186f8e9110"}
Oct 30 22:46:52.058297 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:46:52.058244    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-smnlr" podStartSLOduration=2.058230716 podStartE2EDuration="2.058230716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:46:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:46:52.057094377 +0000 UTC m=+3389.661401934" watchObservedRunningTime="2026-10-30 22:46:52.058230716 +0000 UTC m=+3389.662538269"
Oct 30 22:46:52.522785 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:46:52.522755    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7470c91-4aae-43b0-a57f-a413090e2321" path="/var/lib/kubelet/pods/f7470c91-4aae-43b0-a57f-a413090e2321/volumes"
Oct 30 22:46:52.953774 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:46:52.953739    5678 generic.go:356] "Generic (PLEG): container finished" podID="c358407d-6272-41ed-8c9a-7a9705d51872" containerID="118df99c26939f10d7f38d8a632d850632d915b5056574c3bcd24ee55becc8b9" exitCode=0
Oct 30 22:46:52.954211 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:46:52.953818    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-smnlr" event={"ID":"c358407d-6272-41ed-8c9a-7a9705d51872","Type":"ContainerDied","Data":"118df99c26939f10d7f38d8a632d850632d915b5056574c3bcd24ee55becc8b9"}
Oct 30 22:46:54.090233 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:46:54.090205    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-smnlr"
Oct 30 22:46:54.230272 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:46:54.230192    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c358407d-6272-41ed-8c9a-7a9705d51872-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c358407d-6272-41ed-8c9a-7a9705d51872-maas-api-xks-ca\") pod \"c358407d-6272-41ed-8c9a-7a9705d51872\" (UID: \"c358407d-6272-41ed-8c9a-7a9705d51872\") "
Oct 30 22:46:54.230272 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:46:54.230240    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c358407d-6272-41ed-8c9a-7a9705d51872-kube-api-access-wzxd4\" (UniqueName: \"kubernetes.io/projected/c358407d-6272-41ed-8c9a-7a9705d51872-kube-api-access-wzxd4\") pod \"c358407d-6272-41ed-8c9a-7a9705d51872\" (UID: \"c358407d-6272-41ed-8c9a-7a9705d51872\") "
Oct 30 22:46:54.230490 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:46:54.230291    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c358407d-6272-41ed-8c9a-7a9705d51872-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c358407d-6272-41ed-8c9a-7a9705d51872-maas-api-openshift-ca\") pod \"c358407d-6272-41ed-8c9a-7a9705d51872\" (UID: \"c358407d-6272-41ed-8c9a-7a9705d51872\") "
Oct 30 22:46:54.230710 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:46:54.230686    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c358407d-6272-41ed-8c9a-7a9705d51872-maas-api-openshift-ca" pod "c358407d-6272-41ed-8c9a-7a9705d51872" (UID: "c358407d-6272-41ed-8c9a-7a9705d51872"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:46:54.232451 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:46:54.232425    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c358407d-6272-41ed-8c9a-7a9705d51872-kube-api-access-wzxd4" pod "c358407d-6272-41ed-8c9a-7a9705d51872" (UID: "c358407d-6272-41ed-8c9a-7a9705d51872"). InnerVolumeSpecName "kube-api-access-wzxd4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:46:54.232451 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:46:54.232433    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c358407d-6272-41ed-8c9a-7a9705d51872-maas-api-xks-ca" pod "c358407d-6272-41ed-8c9a-7a9705d51872" (UID: "c358407d-6272-41ed-8c9a-7a9705d51872"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:46:54.332075 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:46:54.332043    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c358407d-6272-41ed-8c9a-7a9705d51872-maas-api-xks-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:46:54.332075 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:46:54.332070    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzxd4\" (UniqueName: \"kubernetes.io/projected/c358407d-6272-41ed-8c9a-7a9705d51872-kube-api-access-wzxd4\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:46:54.332253 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:46:54.332086    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c358407d-6272-41ed-8c9a-7a9705d51872-maas-api-openshift-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:46:54.965453 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:46:54.965425    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-smnlr"
Oct 30 22:46:54.965626 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:46:54.965423    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-smnlr" event={"ID":"c358407d-6272-41ed-8c9a-7a9705d51872","Type":"ContainerDied","Data":"cfd447c60c251c5c775f5258b93b45f1d8c386b057ad550d2edbd6186f8e9110"}
Oct 30 22:46:54.965626 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:46:54.965543    5678 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cfd447c60c251c5c775f5258b93b45f1d8c386b057ad550d2edbd6186f8e9110"
Oct 30 22:47:10.570639 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:47:10.570601    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 22:47:24.388517 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:47:24.388483    5678 scope.go:122] "RemoveContainer" containerID="da32e171a3f0f4368b920fc6c78407b32d0afeb9afdb479f542781e17d083b63"
Oct 30 22:47:24.397343 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:47:24.397323    5678 scope.go:122] "RemoveContainer" containerID="4504d45f1e8b356ed49218e41c9482c91399a1fffe65b821389939a763c60889"
Oct 30 22:47:24.404887 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:47:24.404865    5678 scope.go:122] "RemoveContainer" containerID="60c78f33c65112c090858ae6419d3003b9b82d40344881d3a2700fb5570ddc59"
Oct 30 22:47:40.570534 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:47:40.570445    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 22:47:41.285632 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:47:41.285600    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-nz85w"]
Oct 30 22:47:41.286049 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:47:41.286032    5678 cpu_manager.go:414] "containerMap: removing container" podUID="c358407d-6272-41ed-8c9a-7a9705d51872" containerName="revoke-keys"
Oct 30 22:47:41.286166 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:47:41.286052    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c358407d-6272-41ed-8c9a-7a9705d51872" containerName="revoke-keys"
Oct 30 22:47:41.286166 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:47:41.286141    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="c358407d-6272-41ed-8c9a-7a9705d51872" containerName="revoke-keys"
Oct 30 22:47:41.291123 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:47:41.291096    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-nz85w"
Oct 30 22:47:41.344207 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:47:41.344174    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-nz85w"]
Oct 30 22:47:41.377751 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:47:41.377719    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2ff7913-4053-42e7-8b66-49ad5b4ab7ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-nz85w\" (UID: \"a2ff7913-4053-42e7-8b66-49ad5b4ab7ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-nz85w"
Oct 30 22:47:41.377905 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:47:41.377773    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fh5n7\" (UniqueName: \"kubernetes.io/projected/a2ff7913-4053-42e7-8b66-49ad5b4ab7ba-kube-api-access-fh5n7\") pod \"maas-api-revoke-sub-e2fc9c5b-nz85w\" (UID: \"a2ff7913-4053-42e7-8b66-49ad5b4ab7ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-nz85w"
Oct 30 22:47:41.377950 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:47:41.377913    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2ff7913-4053-42e7-8b66-49ad5b4ab7ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-nz85w\" (UID: \"a2ff7913-4053-42e7-8b66-49ad5b4ab7ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-nz85w"
Oct 30 22:47:41.388565 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:47:41.388534    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:47:41.388705 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:47:41.388595    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:47:41.388777 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:47:41.388745    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5rvcj\""
Oct 30 22:47:41.388777 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:47:41.388746    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:47:41.479107 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:47:41.479066    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fh5n7\" (UniqueName: \"kubernetes.io/projected/a2ff7913-4053-42e7-8b66-49ad5b4ab7ba-kube-api-access-fh5n7\") pod \"maas-api-revoke-sub-e2fc9c5b-nz85w\" (UID: \"a2ff7913-4053-42e7-8b66-49ad5b4ab7ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-nz85w"
Oct 30 22:47:41.479301 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:47:41.479161    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2ff7913-4053-42e7-8b66-49ad5b4ab7ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-nz85w\" (UID: \"a2ff7913-4053-42e7-8b66-49ad5b4ab7ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-nz85w"
Oct 30 22:47:41.479301 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:47:41.479198    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2ff7913-4053-42e7-8b66-49ad5b4ab7ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-nz85w\" (UID: \"a2ff7913-4053-42e7-8b66-49ad5b4ab7ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-nz85w"
Oct 30 22:47:41.479840 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:47:41.479815    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2ff7913-4053-42e7-8b66-49ad5b4ab7ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-nz85w\" (UID: \"a2ff7913-4053-42e7-8b66-49ad5b4ab7ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-nz85w"
Oct 30 22:47:41.481467 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:47:41.481441    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2ff7913-4053-42e7-8b66-49ad5b4ab7ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-nz85w\" (UID: \"a2ff7913-4053-42e7-8b66-49ad5b4ab7ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-nz85w"
Oct 30 22:47:41.537620 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:47:41.537563    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fh5n7\" (UniqueName: \"kubernetes.io/projected/a2ff7913-4053-42e7-8b66-49ad5b4ab7ba-kube-api-access-fh5n7\") pod \"maas-api-revoke-sub-e2fc9c5b-nz85w\" (UID: \"a2ff7913-4053-42e7-8b66-49ad5b4ab7ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-nz85w"
Oct 30 22:47:41.607182 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:47:41.607153    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-nz85w"
Oct 30 22:47:41.987713 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:47:41.987665    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-nz85w"]
Oct 30 22:47:41.994551 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:47:41.994512    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2ff7913_4053_42e7_8b66_49ad5b4ab7ba.slice/crio-1803e05a2117569c8892a53a9daecb94b53036c50cf555e9116a9f50ec10711d WatchSource:0}: Error finding container 1803e05a2117569c8892a53a9daecb94b53036c50cf555e9116a9f50ec10711d: Status 404 returned error can't find the container with id 1803e05a2117569c8892a53a9daecb94b53036c50cf555e9116a9f50ec10711d
Oct 30 22:47:42.249454 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:47:42.249364    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-nz85w" event={"ID":"a2ff7913-4053-42e7-8b66-49ad5b4ab7ba","Type":"ContainerStarted","Data":"149cf47f78cb0cf819b29adad90663f14fc28ea0844f6eda6abe719febd3a3be"}
Oct 30 22:47:42.249454 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:47:42.249444    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-nz85w" event={"ID":"a2ff7913-4053-42e7-8b66-49ad5b4ab7ba","Type":"ContainerStarted","Data":"1803e05a2117569c8892a53a9daecb94b53036c50cf555e9116a9f50ec10711d"}
Oct 30 22:47:42.361412 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:47:42.361351    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-nz85w" podStartSLOduration=1.361338287 podStartE2EDuration="1.361338287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:47:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:47:42.360175191 +0000 UTC m=+3439.964482744" watchObservedRunningTime="2026-10-30 22:47:42.361338287 +0000 UTC m=+3439.965645840"
Oct 30 22:47:43.255711 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:47:43.255676    5678 generic.go:356] "Generic (PLEG): container finished" podID="a2ff7913-4053-42e7-8b66-49ad5b4ab7ba" containerID="149cf47f78cb0cf819b29adad90663f14fc28ea0844f6eda6abe719febd3a3be" exitCode=0
Oct 30 22:47:43.256131 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:47:43.255757    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-nz85w" event={"ID":"a2ff7913-4053-42e7-8b66-49ad5b4ab7ba","Type":"ContainerDied","Data":"149cf47f78cb0cf819b29adad90663f14fc28ea0844f6eda6abe719febd3a3be"}
Oct 30 22:47:44.397592 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:47:44.397569    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-nz85w"
Oct 30 22:47:44.506782 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:47:44.506751    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2ff7913-4053-42e7-8b66-49ad5b4ab7ba-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2ff7913-4053-42e7-8b66-49ad5b4ab7ba-maas-api-xks-ca\") pod \"a2ff7913-4053-42e7-8b66-49ad5b4ab7ba\" (UID: \"a2ff7913-4053-42e7-8b66-49ad5b4ab7ba\") "
Oct 30 22:47:44.506960 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:47:44.506801    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2ff7913-4053-42e7-8b66-49ad5b4ab7ba-kube-api-access-fh5n7\" (UniqueName: \"kubernetes.io/projected/a2ff7913-4053-42e7-8b66-49ad5b4ab7ba-kube-api-access-fh5n7\") pod \"a2ff7913-4053-42e7-8b66-49ad5b4ab7ba\" (UID: \"a2ff7913-4053-42e7-8b66-49ad5b4ab7ba\") "
Oct 30 22:47:44.506960 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:47:44.506934    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2ff7913-4053-42e7-8b66-49ad5b4ab7ba-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2ff7913-4053-42e7-8b66-49ad5b4ab7ba-maas-api-openshift-ca\") pod \"a2ff7913-4053-42e7-8b66-49ad5b4ab7ba\" (UID: \"a2ff7913-4053-42e7-8b66-49ad5b4ab7ba\") "
Oct 30 22:47:44.507308 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:47:44.507278    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2ff7913-4053-42e7-8b66-49ad5b4ab7ba-maas-api-openshift-ca" pod "a2ff7913-4053-42e7-8b66-49ad5b4ab7ba" (UID: "a2ff7913-4053-42e7-8b66-49ad5b4ab7ba"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:47:44.508982 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:47:44.508933    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2ff7913-4053-42e7-8b66-49ad5b4ab7ba-maas-api-xks-ca" pod "a2ff7913-4053-42e7-8b66-49ad5b4ab7ba" (UID: "a2ff7913-4053-42e7-8b66-49ad5b4ab7ba"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:47:44.509171 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:47:44.509146    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2ff7913-4053-42e7-8b66-49ad5b4ab7ba-kube-api-access-fh5n7" pod "a2ff7913-4053-42e7-8b66-49ad5b4ab7ba" (UID: "a2ff7913-4053-42e7-8b66-49ad5b4ab7ba"). InnerVolumeSpecName "kube-api-access-fh5n7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:47:44.608284 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:47:44.608257    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fh5n7\" (UniqueName: \"kubernetes.io/projected/a2ff7913-4053-42e7-8b66-49ad5b4ab7ba-kube-api-access-fh5n7\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:44.608284 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:47:44.608282    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2ff7913-4053-42e7-8b66-49ad5b4ab7ba-maas-api-openshift-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:44.608457 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:47:44.608291    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2ff7913-4053-42e7-8b66-49ad5b4ab7ba-maas-api-xks-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:45.047773 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:47:45.047733    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-5rv82"]
Oct 30 22:47:45.094799 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:47:45.094714    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-5rv82"]
Oct 30 22:47:45.267044 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:47:45.267006    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-nz85w" event={"ID":"a2ff7913-4053-42e7-8b66-49ad5b4ab7ba","Type":"ContainerDied","Data":"1803e05a2117569c8892a53a9daecb94b53036c50cf555e9116a9f50ec10711d"}
Oct 30 22:47:45.267044 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:47:45.267035    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-nz85w"
Oct 30 22:47:45.267248 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:47:45.267063    5678 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1803e05a2117569c8892a53a9daecb94b53036c50cf555e9116a9f50ec10711d"
Oct 30 22:47:46.521363 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:47:46.521330    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6140d0a-f4e4-4021-ad69-32ea10e389ff" path="/var/lib/kubelet/pods/e6140d0a-f4e4-4021-ad69-32ea10e389ff/volumes"
Oct 30 22:47:52.061955 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:47:52.061926    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-rmspf"]
Oct 30 22:47:52.062365 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:47:52.062350    5678 cpu_manager.go:414] "containerMap: removing container" podUID="a2ff7913-4053-42e7-8b66-49ad5b4ab7ba" containerName="revoke-keys"
Oct 30 22:47:52.062427 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:47:52.062366    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2ff7913-4053-42e7-8b66-49ad5b4ab7ba" containerName="revoke-keys"
Oct 30 22:47:52.062463 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:47:52.062432    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2ff7913-4053-42e7-8b66-49ad5b4ab7ba" containerName="revoke-keys"
Oct 30 22:47:52.065961 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:47:52.065942    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-rmspf"
Oct 30 22:47:52.118283 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:47:52.118250    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:47:52.165391 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:47:52.165362    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:47:52.165677 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:47:52.165648    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:47:52.165793 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:47:52.165697    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5rvcj\""
Oct 30 22:47:52.167893 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:47:52.167876    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-rmspf"]
Oct 30 22:47:52.178194 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:47:52.178167    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ce8b029-1e6a-42e0-a0af-37e067c7a618-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-rmspf\" (UID: \"5ce8b029-1e6a-42e0-a0af-37e067c7a618\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-rmspf"
Oct 30 22:47:52.178306 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:47:52.178202    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79422\" (UniqueName: \"kubernetes.io/projected/5ce8b029-1e6a-42e0-a0af-37e067c7a618-kube-api-access-79422\") pod \"maas-api-revoke-sub-5a29efb3-rmspf\" (UID: \"5ce8b029-1e6a-42e0-a0af-37e067c7a618\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-rmspf"
Oct 30 22:47:52.178306 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:47:52.178229    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ce8b029-1e6a-42e0-a0af-37e067c7a618-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-rmspf\" (UID: \"5ce8b029-1e6a-42e0-a0af-37e067c7a618\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-rmspf"
Oct 30 22:47:52.278956 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:47:52.278926    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ce8b029-1e6a-42e0-a0af-37e067c7a618-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-rmspf\" (UID: \"5ce8b029-1e6a-42e0-a0af-37e067c7a618\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-rmspf"
Oct 30 22:47:52.278956 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:47:52.278960    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79422\" (UniqueName: \"kubernetes.io/projected/5ce8b029-1e6a-42e0-a0af-37e067c7a618-kube-api-access-79422\") pod \"maas-api-revoke-sub-5a29efb3-rmspf\" (UID: \"5ce8b029-1e6a-42e0-a0af-37e067c7a618\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-rmspf"
Oct 30 22:47:52.279187 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:47:52.278979    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ce8b029-1e6a-42e0-a0af-37e067c7a618-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-rmspf\" (UID: \"5ce8b029-1e6a-42e0-a0af-37e067c7a618\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-rmspf"
Oct 30 22:47:52.279721 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:47:52.279698    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ce8b029-1e6a-42e0-a0af-37e067c7a618-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-rmspf\" (UID: \"5ce8b029-1e6a-42e0-a0af-37e067c7a618\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-rmspf"
Oct 30 22:47:52.281329 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:47:52.281310    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ce8b029-1e6a-42e0-a0af-37e067c7a618-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-rmspf\" (UID: \"5ce8b029-1e6a-42e0-a0af-37e067c7a618\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-rmspf"
Oct 30 22:47:52.337852 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:47:52.337795    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79422\" (UniqueName: \"kubernetes.io/projected/5ce8b029-1e6a-42e0-a0af-37e067c7a618-kube-api-access-79422\") pod \"maas-api-revoke-sub-5a29efb3-rmspf\" (UID: \"5ce8b029-1e6a-42e0-a0af-37e067c7a618\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-rmspf"
Oct 30 22:47:52.378591 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:47:52.378571    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-rmspf"
Oct 30 22:47:52.762261 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:47:52.762228    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-rmspf"]
Oct 30 22:47:52.769020 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:47:52.768985    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ce8b029_1e6a_42e0_a0af_37e067c7a618.slice/crio-859b99e1edb9e3725bf06589bbf442f5d1d8d7b16d8a2d041bdadb754b1ca85c WatchSource:0}: Error finding container 859b99e1edb9e3725bf06589bbf442f5d1d8d7b16d8a2d041bdadb754b1ca85c: Status 404 returned error can't find the container with id 859b99e1edb9e3725bf06589bbf442f5d1d8d7b16d8a2d041bdadb754b1ca85c
Oct 30 22:47:53.310672 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:47:53.310596    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-rmspf" event={"ID":"5ce8b029-1e6a-42e0-a0af-37e067c7a618","Type":"ContainerStarted","Data":"1d4c3f291f44c6322db2cb8fa7e8fb11ba0d14eb3641909430d7bffe914003e5"}
Oct 30 22:47:53.310672 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:47:53.310651    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-rmspf" event={"ID":"5ce8b029-1e6a-42e0-a0af-37e067c7a618","Type":"ContainerStarted","Data":"859b99e1edb9e3725bf06589bbf442f5d1d8d7b16d8a2d041bdadb754b1ca85c"}
Oct 30 22:47:53.423532 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:47:53.423469    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-rmspf" podStartSLOduration=1.423452518 podStartE2EDuration="1.423452518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:47:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:47:53.422174956 +0000 UTC m=+3451.026482509" watchObservedRunningTime="2026-10-30 22:47:53.423452518 +0000 UTC m=+3451.027760074"
Oct 30 22:47:54.317721 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:47:54.317685    5678 generic.go:356] "Generic (PLEG): container finished" podID="5ce8b029-1e6a-42e0-a0af-37e067c7a618" containerID="1d4c3f291f44c6322db2cb8fa7e8fb11ba0d14eb3641909430d7bffe914003e5" exitCode=0
Oct 30 22:47:54.318124 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:47:54.317737    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-rmspf" event={"ID":"5ce8b029-1e6a-42e0-a0af-37e067c7a618","Type":"ContainerDied","Data":"1d4c3f291f44c6322db2cb8fa7e8fb11ba0d14eb3641909430d7bffe914003e5"}
Oct 30 22:47:55.680455 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:47:55.675706    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-rmspf"
Oct 30 22:47:55.713997 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:47:55.713949    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ce8b029-1e6a-42e0-a0af-37e067c7a618-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ce8b029-1e6a-42e0-a0af-37e067c7a618-maas-api-xks-ca\") pod \"5ce8b029-1e6a-42e0-a0af-37e067c7a618\" (UID: \"5ce8b029-1e6a-42e0-a0af-37e067c7a618\") "
Oct 30 22:47:55.714178 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:47:55.714056    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ce8b029-1e6a-42e0-a0af-37e067c7a618-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ce8b029-1e6a-42e0-a0af-37e067c7a618-maas-api-openshift-ca\") pod \"5ce8b029-1e6a-42e0-a0af-37e067c7a618\" (UID: \"5ce8b029-1e6a-42e0-a0af-37e067c7a618\") "
Oct 30 22:47:55.714178 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:47:55.714075    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ce8b029-1e6a-42e0-a0af-37e067c7a618-kube-api-access-79422\" (UniqueName: \"kubernetes.io/projected/5ce8b029-1e6a-42e0-a0af-37e067c7a618-kube-api-access-79422\") pod \"5ce8b029-1e6a-42e0-a0af-37e067c7a618\" (UID: \"5ce8b029-1e6a-42e0-a0af-37e067c7a618\") "
Oct 30 22:47:55.714736 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:47:55.714702    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ce8b029-1e6a-42e0-a0af-37e067c7a618-maas-api-openshift-ca" pod "5ce8b029-1e6a-42e0-a0af-37e067c7a618" (UID: "5ce8b029-1e6a-42e0-a0af-37e067c7a618"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:47:55.716431 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:47:55.716388    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ce8b029-1e6a-42e0-a0af-37e067c7a618-maas-api-xks-ca" pod "5ce8b029-1e6a-42e0-a0af-37e067c7a618" (UID: "5ce8b029-1e6a-42e0-a0af-37e067c7a618"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:47:55.716431 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:47:55.716422    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ce8b029-1e6a-42e0-a0af-37e067c7a618-kube-api-access-79422" pod "5ce8b029-1e6a-42e0-a0af-37e067c7a618" (UID: "5ce8b029-1e6a-42e0-a0af-37e067c7a618"). InnerVolumeSpecName "kube-api-access-79422". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:47:55.815453 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:47:55.815395    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ce8b029-1e6a-42e0-a0af-37e067c7a618-maas-api-openshift-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:55.815453 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:47:55.815446    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79422\" (UniqueName: \"kubernetes.io/projected/5ce8b029-1e6a-42e0-a0af-37e067c7a618-kube-api-access-79422\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:55.815453 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:47:55.815456    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ce8b029-1e6a-42e0-a0af-37e067c7a618-maas-api-xks-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:56.384041 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:47:56.384001    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-rmspf" event={"ID":"5ce8b029-1e6a-42e0-a0af-37e067c7a618","Type":"ContainerDied","Data":"859b99e1edb9e3725bf06589bbf442f5d1d8d7b16d8a2d041bdadb754b1ca85c"}
Oct 30 22:47:56.384320 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:47:56.384300    5678 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="859b99e1edb9e3725bf06589bbf442f5d1d8d7b16d8a2d041bdadb754b1ca85c"
Oct 30 22:47:56.384523 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:47:56.384507    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-rmspf"
Oct 30 22:48:10.573701 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:48:10.570893    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 22:48:12.454655 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:48:12.454615    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-smnlr"]
Oct 30 22:48:12.501710 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:48:12.501681    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-smnlr"]
Oct 30 22:48:12.522023 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:48:12.521989    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c358407d-6272-41ed-8c9a-7a9705d51872" path="/var/lib/kubelet/pods/c358407d-6272-41ed-8c9a-7a9705d51872/volumes"
Oct 30 22:48:12.580383 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:48:12.580354    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fxhkk"]
Oct 30 22:48:12.580743 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:48:12.580730    5678 cpu_manager.go:414] "containerMap: removing container" podUID="5ce8b029-1e6a-42e0-a0af-37e067c7a618" containerName="revoke-keys"
Oct 30 22:48:12.580787 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:48:12.580744    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ce8b029-1e6a-42e0-a0af-37e067c7a618" containerName="revoke-keys"
Oct 30 22:48:12.580828 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:48:12.580819    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ce8b029-1e6a-42e0-a0af-37e067c7a618" containerName="revoke-keys"
Oct 30 22:48:12.584028 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:48:12.584008    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fxhkk"
Oct 30 22:48:12.636095 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:48:12.636049    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:48:12.636245 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:48:12.636118    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:48:12.636245 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:48:12.636197    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5rvcj\""
Oct 30 22:48:12.636245 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:48:12.636235    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:48:12.638555 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:48:12.638529    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fxhkk"]
Oct 30 22:48:12.679261 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:48:12.679231    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa4525e5-59d9-413f-9f38-be7b1f37a359-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-fxhkk\" (UID: \"fa4525e5-59d9-413f-9f38-be7b1f37a359\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fxhkk"
Oct 30 22:48:12.679383 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:48:12.679299    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa4525e5-59d9-413f-9f38-be7b1f37a359-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-fxhkk\" (UID: \"fa4525e5-59d9-413f-9f38-be7b1f37a359\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fxhkk"
Oct 30 22:48:12.679383 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:48:12.679370    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsnh4\" (UniqueName: \"kubernetes.io/projected/fa4525e5-59d9-413f-9f38-be7b1f37a359-kube-api-access-gsnh4\") pod \"maas-api-revoke-sub-3d4156f1-fxhkk\" (UID: \"fa4525e5-59d9-413f-9f38-be7b1f37a359\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fxhkk"
Oct 30 22:48:12.780518 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:48:12.780440    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa4525e5-59d9-413f-9f38-be7b1f37a359-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-fxhkk\" (UID: \"fa4525e5-59d9-413f-9f38-be7b1f37a359\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fxhkk"
Oct 30 22:48:12.780518 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:48:12.780499    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa4525e5-59d9-413f-9f38-be7b1f37a359-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-fxhkk\" (UID: \"fa4525e5-59d9-413f-9f38-be7b1f37a359\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fxhkk"
Oct 30 22:48:12.780706 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:48:12.780538    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsnh4\" (UniqueName: \"kubernetes.io/projected/fa4525e5-59d9-413f-9f38-be7b1f37a359-kube-api-access-gsnh4\") pod \"maas-api-revoke-sub-3d4156f1-fxhkk\" (UID: \"fa4525e5-59d9-413f-9f38-be7b1f37a359\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fxhkk"
Oct 30 22:48:12.781207 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:48:12.781187    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa4525e5-59d9-413f-9f38-be7b1f37a359-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-fxhkk\" (UID: \"fa4525e5-59d9-413f-9f38-be7b1f37a359\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fxhkk"
Oct 30 22:48:12.782706 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:48:12.782686    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa4525e5-59d9-413f-9f38-be7b1f37a359-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-fxhkk\" (UID: \"fa4525e5-59d9-413f-9f38-be7b1f37a359\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fxhkk"
Oct 30 22:48:12.839065 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:48:12.839038    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsnh4\" (UniqueName: \"kubernetes.io/projected/fa4525e5-59d9-413f-9f38-be7b1f37a359-kube-api-access-gsnh4\") pod \"maas-api-revoke-sub-3d4156f1-fxhkk\" (UID: \"fa4525e5-59d9-413f-9f38-be7b1f37a359\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fxhkk"
Oct 30 22:48:12.896509 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:48:12.896481    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fxhkk"
Oct 30 22:48:13.276837 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:48:13.276805    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fxhkk"]
Oct 30 22:48:13.284280 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:48:13.284247    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa4525e5_59d9_413f_9f38_be7b1f37a359.slice/crio-a6f11a6c96628ae47c407df56858d7ae603627ad69d5b212f92d5b9e09c29de5 WatchSource:0}: Error finding container a6f11a6c96628ae47c407df56858d7ae603627ad69d5b212f92d5b9e09c29de5: Status 404 returned error can't find the container with id a6f11a6c96628ae47c407df56858d7ae603627ad69d5b212f92d5b9e09c29de5
Oct 30 22:48:13.473670 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:48:13.473641    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fxhkk" event={"ID":"fa4525e5-59d9-413f-9f38-be7b1f37a359","Type":"ContainerStarted","Data":"a6f11a6c96628ae47c407df56858d7ae603627ad69d5b212f92d5b9e09c29de5"}
Oct 30 22:48:14.480694 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:48:14.480614    5678 generic.go:356] "Generic (PLEG): container finished" podID="fa4525e5-59d9-413f-9f38-be7b1f37a359" containerID="3b37b82e8475780de45165bc40d6eb6400a0d0b18dac0bebd4be3048a7071158" exitCode=0
Oct 30 22:48:14.481064 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:48:14.480701    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fxhkk" event={"ID":"fa4525e5-59d9-413f-9f38-be7b1f37a359","Type":"ContainerDied","Data":"3b37b82e8475780de45165bc40d6eb6400a0d0b18dac0bebd4be3048a7071158"}
Oct 30 22:48:15.615886 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:48:15.615862    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fxhkk"
Oct 30 22:48:15.706529 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:48:15.706499    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa4525e5-59d9-413f-9f38-be7b1f37a359-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa4525e5-59d9-413f-9f38-be7b1f37a359-maas-api-openshift-ca\") pod \"fa4525e5-59d9-413f-9f38-be7b1f37a359\" (UID: \"fa4525e5-59d9-413f-9f38-be7b1f37a359\") "
Oct 30 22:48:15.706695 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:48:15.706586    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa4525e5-59d9-413f-9f38-be7b1f37a359-kube-api-access-gsnh4\" (UniqueName: \"kubernetes.io/projected/fa4525e5-59d9-413f-9f38-be7b1f37a359-kube-api-access-gsnh4\") pod \"fa4525e5-59d9-413f-9f38-be7b1f37a359\" (UID: \"fa4525e5-59d9-413f-9f38-be7b1f37a359\") "
Oct 30 22:48:15.706741 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:48:15.706684    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa4525e5-59d9-413f-9f38-be7b1f37a359-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa4525e5-59d9-413f-9f38-be7b1f37a359-maas-api-xks-ca\") pod \"fa4525e5-59d9-413f-9f38-be7b1f37a359\" (UID: \"fa4525e5-59d9-413f-9f38-be7b1f37a359\") "
Oct 30 22:48:15.706914 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:48:15.706882    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa4525e5-59d9-413f-9f38-be7b1f37a359-maas-api-openshift-ca" pod "fa4525e5-59d9-413f-9f38-be7b1f37a359" (UID: "fa4525e5-59d9-413f-9f38-be7b1f37a359"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:48:15.707053 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:48:15.707030    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa4525e5-59d9-413f-9f38-be7b1f37a359-maas-api-openshift-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:48:15.708664 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:48:15.708639    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa4525e5-59d9-413f-9f38-be7b1f37a359-kube-api-access-gsnh4" pod "fa4525e5-59d9-413f-9f38-be7b1f37a359" (UID: "fa4525e5-59d9-413f-9f38-be7b1f37a359"). InnerVolumeSpecName "kube-api-access-gsnh4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:48:15.708664 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:48:15.708644    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa4525e5-59d9-413f-9f38-be7b1f37a359-maas-api-xks-ca" pod "fa4525e5-59d9-413f-9f38-be7b1f37a359" (UID: "fa4525e5-59d9-413f-9f38-be7b1f37a359"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:48:15.808530 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:48:15.808449    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsnh4\" (UniqueName: \"kubernetes.io/projected/fa4525e5-59d9-413f-9f38-be7b1f37a359-kube-api-access-gsnh4\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:48:15.808530 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:48:15.808477    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa4525e5-59d9-413f-9f38-be7b1f37a359-maas-api-xks-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:48:16.048179 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:48:16.048145    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-qpfrl"]
Oct 30 22:48:16.053425 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:48:16.053376    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-qpfrl"]
Oct 30 22:48:16.492789 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:48:16.492757    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fxhkk"
Oct 30 22:48:16.492991 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:48:16.492802    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fxhkk" event={"ID":"fa4525e5-59d9-413f-9f38-be7b1f37a359","Type":"ContainerDied","Data":"a6f11a6c96628ae47c407df56858d7ae603627ad69d5b212f92d5b9e09c29de5"}
Oct 30 22:48:16.492991 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:48:16.492886    5678 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a6f11a6c96628ae47c407df56858d7ae603627ad69d5b212f92d5b9e09c29de5"
Oct 30 22:48:16.521272 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:48:16.521246    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f95d6797-0031-47cd-acf4-e7a7e41943ea" path="/var/lib/kubelet/pods/f95d6797-0031-47cd-acf4-e7a7e41943ea/volumes"
Oct 30 22:48:24.436764 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:48:24.436734    5678 scope.go:122] "RemoveContainer" containerID="83195f87d44d9899f034267dc33d3de33ebfe7ce9da7695320024dda82b76c7f"
Oct 30 22:48:29.056424 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:48:29.055584    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-r72qh"]
Oct 30 22:48:29.094773 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:48:29.094732    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-r72qh"]
Oct 30 22:48:30.522175 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:48:30.522136    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c05d3b91-fbb9-43b5-8837-994668e18f51" path="/var/lib/kubelet/pods/c05d3b91-fbb9-43b5-8837-994668e18f51/volumes"
Oct 30 22:48:40.572745 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:48:40.572640    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 22:49:04.047790 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:49:04.047716    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-f62pp"]
Oct 30 22:49:04.094570 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:49:04.094541    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-f62pp"]
Oct 30 22:49:04.522138 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:49:04.522107    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da43d8a5-64f1-4f97-9841-4f86d7e23cdb" path="/var/lib/kubelet/pods/da43d8a5-64f1-4f97-9841-4f86d7e23cdb/volumes"
Oct 30 22:49:10.571472 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:49:10.571372    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 22:49:15.066173 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:49:15.066138    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-2r5j6"]
Oct 30 22:49:15.112949 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:49:15.112919    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-2r5j6"]
Oct 30 22:49:16.522150 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:49:16.522117    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0664869-236d-4245-8d16-374b217dda8d" path="/var/lib/kubelet/pods/c0664869-236d-4245-8d16-374b217dda8d/volumes"
Oct 30 22:49:24.459264 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:49:24.459231    5678 scope.go:122] "RemoveContainer" containerID="aa34951ec31137690f30f22275e248b6d78cb620bd135c2342371e50905e1125"
Oct 30 22:49:24.467607 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:49:24.467587    5678 scope.go:122] "RemoveContainer" containerID="e72d11bd3cd5b67cf5eb788f6faffee2d78b913f8ffae9f80fc92956d3cabc4b"
Oct 30 22:49:40.570964 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:49:40.570926    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 22:49:50.048516 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:49:50.048481    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-n7chv"]
Oct 30 22:49:50.095882 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:49:50.095854    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-n7chv"]
Oct 30 22:49:50.522580 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:49:50.522540    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4afea247-e8e1-455e-ae19-6e9c4282ba84" path="/var/lib/kubelet/pods/4afea247-e8e1-455e-ae19-6e9c4282ba84/volumes"
Oct 30 22:49:57.265251 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:49:57.265212    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78019b2d-vqqll"]
Oct 30 22:49:57.268578 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:49:57.266545    5678 cpu_manager.go:414] "containerMap: removing container" podUID="fa4525e5-59d9-413f-9f38-be7b1f37a359" containerName="revoke-keys"
Oct 30 22:49:57.268578 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:49:57.266576    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa4525e5-59d9-413f-9f38-be7b1f37a359" containerName="revoke-keys"
Oct 30 22:49:57.268578 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:49:57.266790    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa4525e5-59d9-413f-9f38-be7b1f37a359" containerName="revoke-keys"
Oct 30 22:49:57.273726 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:49:57.273701    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78019b2d-vqqll"
Oct 30 22:49:57.313969 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:49:57.313943    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d78034ad-6371-41ef-bb44-f002da98a5a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-78019b2d-vqqll\" (UID: \"d78034ad-6371-41ef-bb44-f002da98a5a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78019b2d-vqqll"
Oct 30 22:49:57.314086 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:49:57.313993    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d78034ad-6371-41ef-bb44-f002da98a5a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-78019b2d-vqqll\" (UID: \"d78034ad-6371-41ef-bb44-f002da98a5a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78019b2d-vqqll"
Oct 30 22:49:57.314086 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:49:57.314056    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftwcw\" (UniqueName: \"kubernetes.io/projected/d78034ad-6371-41ef-bb44-f002da98a5a5-kube-api-access-ftwcw\") pod \"maas-api-revoke-sub-78019b2d-vqqll\" (UID: \"d78034ad-6371-41ef-bb44-f002da98a5a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78019b2d-vqqll"
Oct 30 22:49:57.324928 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:49:57.324905    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78019b2d-vqqll"]
Oct 30 22:49:57.369778 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:49:57.369748    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:49:57.369778 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:49:57.369765    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:49:57.370666 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:49:57.370648    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5rvcj\""
Oct 30 22:49:57.370780 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:49:57.370693    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:49:57.414551 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:49:57.414529    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d78034ad-6371-41ef-bb44-f002da98a5a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-78019b2d-vqqll\" (UID: \"d78034ad-6371-41ef-bb44-f002da98a5a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78019b2d-vqqll"
Oct 30 22:49:57.414699 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:49:57.414577    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d78034ad-6371-41ef-bb44-f002da98a5a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-78019b2d-vqqll\" (UID: \"d78034ad-6371-41ef-bb44-f002da98a5a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78019b2d-vqqll"
Oct 30 22:49:57.414699 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:49:57.414601    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftwcw\" (UniqueName: \"kubernetes.io/projected/d78034ad-6371-41ef-bb44-f002da98a5a5-kube-api-access-ftwcw\") pod \"maas-api-revoke-sub-78019b2d-vqqll\" (UID: \"d78034ad-6371-41ef-bb44-f002da98a5a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78019b2d-vqqll"
Oct 30 22:49:57.415209 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:49:57.415186    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d78034ad-6371-41ef-bb44-f002da98a5a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-78019b2d-vqqll\" (UID: \"d78034ad-6371-41ef-bb44-f002da98a5a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78019b2d-vqqll"
Oct 30 22:49:57.417019 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:49:57.416992    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d78034ad-6371-41ef-bb44-f002da98a5a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-78019b2d-vqqll\" (UID: \"d78034ad-6371-41ef-bb44-f002da98a5a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78019b2d-vqqll"
Oct 30 22:49:57.472768 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:49:57.472740    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftwcw\" (UniqueName: \"kubernetes.io/projected/d78034ad-6371-41ef-bb44-f002da98a5a5-kube-api-access-ftwcw\") pod \"maas-api-revoke-sub-78019b2d-vqqll\" (UID: \"d78034ad-6371-41ef-bb44-f002da98a5a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78019b2d-vqqll"
Oct 30 22:49:57.586812 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:49:57.586745    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78019b2d-vqqll"
Oct 30 22:49:57.966780 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:49:57.966748    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78019b2d-vqqll"]
Oct 30 22:49:57.974189 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:49:57.974156    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd78034ad_6371_41ef_bb44_f002da98a5a5.slice/crio-562039bb9c2f3e6df78c22c0359a76d769b79d28c72987ea93aa3b24f10523f4 WatchSource:0}: Error finding container 562039bb9c2f3e6df78c22c0359a76d769b79d28c72987ea93aa3b24f10523f4: Status 404 returned error can't find the container with id 562039bb9c2f3e6df78c22c0359a76d769b79d28c72987ea93aa3b24f10523f4
Oct 30 22:49:58.063370 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:49:58.063342    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78019b2d-vqqll" event={"ID":"d78034ad-6371-41ef-bb44-f002da98a5a5","Type":"ContainerStarted","Data":"562039bb9c2f3e6df78c22c0359a76d769b79d28c72987ea93aa3b24f10523f4"}
Oct 30 22:49:59.069621 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:49:59.069585    5678 generic.go:356] "Generic (PLEG): container finished" podID="d78034ad-6371-41ef-bb44-f002da98a5a5" containerID="21699f6bce6d2850c8bac5207c485780307ff4a292e2a065e151fd07abd2f98c" exitCode=0
Oct 30 22:49:59.070000 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:49:59.069668    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78019b2d-vqqll" event={"ID":"d78034ad-6371-41ef-bb44-f002da98a5a5","Type":"ContainerDied","Data":"21699f6bce6d2850c8bac5207c485780307ff4a292e2a065e151fd07abd2f98c"}
Oct 30 22:50:00.207009 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:50:00.206981    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78019b2d-vqqll"
Oct 30 22:50:00.239246 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:50:00.239210    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d78034ad-6371-41ef-bb44-f002da98a5a5-kube-api-access-ftwcw\" (UniqueName: \"kubernetes.io/projected/d78034ad-6371-41ef-bb44-f002da98a5a5-kube-api-access-ftwcw\") pod \"d78034ad-6371-41ef-bb44-f002da98a5a5\" (UID: \"d78034ad-6371-41ef-bb44-f002da98a5a5\") "
Oct 30 22:50:00.239431 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:50:00.239274    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d78034ad-6371-41ef-bb44-f002da98a5a5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d78034ad-6371-41ef-bb44-f002da98a5a5-maas-api-openshift-ca\") pod \"d78034ad-6371-41ef-bb44-f002da98a5a5\" (UID: \"d78034ad-6371-41ef-bb44-f002da98a5a5\") "
Oct 30 22:50:00.239431 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:50:00.239314    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d78034ad-6371-41ef-bb44-f002da98a5a5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d78034ad-6371-41ef-bb44-f002da98a5a5-maas-api-xks-ca\") pod \"d78034ad-6371-41ef-bb44-f002da98a5a5\" (UID: \"d78034ad-6371-41ef-bb44-f002da98a5a5\") "
Oct 30 22:50:00.239668 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:50:00.239644    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d78034ad-6371-41ef-bb44-f002da98a5a5-maas-api-openshift-ca" pod "d78034ad-6371-41ef-bb44-f002da98a5a5" (UID: "d78034ad-6371-41ef-bb44-f002da98a5a5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:50:00.241415 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:50:00.241375    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d78034ad-6371-41ef-bb44-f002da98a5a5-maas-api-xks-ca" pod "d78034ad-6371-41ef-bb44-f002da98a5a5" (UID: "d78034ad-6371-41ef-bb44-f002da98a5a5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:50:00.241540 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:50:00.241429    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d78034ad-6371-41ef-bb44-f002da98a5a5-kube-api-access-ftwcw" pod "d78034ad-6371-41ef-bb44-f002da98a5a5" (UID: "d78034ad-6371-41ef-bb44-f002da98a5a5"). InnerVolumeSpecName "kube-api-access-ftwcw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:50:00.340549 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:50:00.340474    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftwcw\" (UniqueName: \"kubernetes.io/projected/d78034ad-6371-41ef-bb44-f002da98a5a5-kube-api-access-ftwcw\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:50:00.340549 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:50:00.340506    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d78034ad-6371-41ef-bb44-f002da98a5a5-maas-api-openshift-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:50:00.340549 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:50:00.340518    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d78034ad-6371-41ef-bb44-f002da98a5a5-maas-api-xks-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:50:01.082163 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:50:01.082136    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78019b2d-vqqll"
Oct 30 22:50:01.082341 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:50:01.082139    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78019b2d-vqqll" event={"ID":"d78034ad-6371-41ef-bb44-f002da98a5a5","Type":"ContainerDied","Data":"562039bb9c2f3e6df78c22c0359a76d769b79d28c72987ea93aa3b24f10523f4"}
Oct 30 22:50:01.082341 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:50:01.082252    5678 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="562039bb9c2f3e6df78c22c0359a76d769b79d28c72987ea93aa3b24f10523f4"
Oct 30 22:50:09.649395 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:50:09.649360    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-88e685-hnxx7"]
Oct 30 22:50:09.649826 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:50:09.649744    5678 cpu_manager.go:414] "containerMap: removing container" podUID="d78034ad-6371-41ef-bb44-f002da98a5a5" containerName="revoke-keys"
Oct 30 22:50:09.649826 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:50:09.649756    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d78034ad-6371-41ef-bb44-f002da98a5a5" containerName="revoke-keys"
Oct 30 22:50:09.649905 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:50:09.649838    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="d78034ad-6371-41ef-bb44-f002da98a5a5" containerName="revoke-keys"
Oct 30 22:50:09.655081 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:50:09.655051    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-88e685-hnxx7"
Oct 30 22:50:09.707651 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:50:09.707615    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-88e685-hnxx7"]
Oct 30 22:50:09.733167 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:50:09.733130    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6599cc7c-79c0-4aab-9f78-206e2b64db4b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-88e685-hnxx7\" (UID: \"6599cc7c-79c0-4aab-9f78-206e2b64db4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-88e685-hnxx7"
Oct 30 22:50:09.733356 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:50:09.733178    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ghq7\" (UniqueName: \"kubernetes.io/projected/6599cc7c-79c0-4aab-9f78-206e2b64db4b-kube-api-access-4ghq7\") pod \"maas-api-revoke-keys-e2e-worker-single-main-88e685-hnxx7\" (UID: \"6599cc7c-79c0-4aab-9f78-206e2b64db4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-88e685-hnxx7"
Oct 30 22:50:09.733356 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:50:09.733265    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6599cc7c-79c0-4aab-9f78-206e2b64db4b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-88e685-hnxx7\" (UID: \"6599cc7c-79c0-4aab-9f78-206e2b64db4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-88e685-hnxx7"
Oct 30 22:50:09.752185 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:50:09.752148    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:50:09.752185 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:50:09.752180    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:50:09.752383 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:50:09.752151    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:50:09.752383 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:50:09.752237    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5rvcj\""
Oct 30 22:50:09.834831 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:50:09.834796    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6599cc7c-79c0-4aab-9f78-206e2b64db4b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-88e685-hnxx7\" (UID: \"6599cc7c-79c0-4aab-9f78-206e2b64db4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-88e685-hnxx7"
Oct 30 22:50:09.835021 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:50:09.834880    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6599cc7c-79c0-4aab-9f78-206e2b64db4b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-88e685-hnxx7\" (UID: \"6599cc7c-79c0-4aab-9f78-206e2b64db4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-88e685-hnxx7"
Oct 30 22:50:09.835021 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:50:09.834970    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ghq7\" (UniqueName: \"kubernetes.io/projected/6599cc7c-79c0-4aab-9f78-206e2b64db4b-kube-api-access-4ghq7\") pod \"maas-api-revoke-keys-e2e-worker-single-main-88e685-hnxx7\" (UID: \"6599cc7c-79c0-4aab-9f78-206e2b64db4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-88e685-hnxx7"
Oct 30 22:50:09.835584 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:50:09.835563    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6599cc7c-79c0-4aab-9f78-206e2b64db4b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-88e685-hnxx7\" (UID: \"6599cc7c-79c0-4aab-9f78-206e2b64db4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-88e685-hnxx7"
Oct 30 22:50:09.837210 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:50:09.837192    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6599cc7c-79c0-4aab-9f78-206e2b64db4b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-88e685-hnxx7\" (UID: \"6599cc7c-79c0-4aab-9f78-206e2b64db4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-88e685-hnxx7"
Oct 30 22:50:09.893686 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:50:09.893649    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ghq7\" (UniqueName: \"kubernetes.io/projected/6599cc7c-79c0-4aab-9f78-206e2b64db4b-kube-api-access-4ghq7\") pod \"maas-api-revoke-keys-e2e-worker-single-main-88e685-hnxx7\" (UID: \"6599cc7c-79c0-4aab-9f78-206e2b64db4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-88e685-hnxx7"
Oct 30 22:50:09.968989 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:50:09.968910    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-88e685-hnxx7"
Oct 30 22:50:10.355122 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:50:10.355029    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-88e685-hnxx7"]
Oct 30 22:50:10.362465 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:50:10.362433    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6599cc7c_79c0_4aab_9f78_206e2b64db4b.slice/crio-1c70f208578e9da9c2146fb79748b83918b7beb9b7de3a9881a2ac2315fdcf11 WatchSource:0}: Error finding container 1c70f208578e9da9c2146fb79748b83918b7beb9b7de3a9881a2ac2315fdcf11: Status 404 returned error can't find the container with id 1c70f208578e9da9c2146fb79748b83918b7beb9b7de3a9881a2ac2315fdcf11
Oct 30 22:50:10.623286 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:50:10.623252    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 22:50:11.135477 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:50:11.135438    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-88e685-hnxx7" event={"ID":"6599cc7c-79c0-4aab-9f78-206e2b64db4b","Type":"ContainerStarted","Data":"c9c115e52235ba0d30e7ae799869f905bc23ac4219f90abe852cb8cc8bae18be"}
Oct 30 22:50:11.135881 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:50:11.135500    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-88e685-hnxx7" event={"ID":"6599cc7c-79c0-4aab-9f78-206e2b64db4b","Type":"ContainerStarted","Data":"1c70f208578e9da9c2146fb79748b83918b7beb9b7de3a9881a2ac2315fdcf11"}
Oct 30 22:50:11.247135 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:50:11.247080    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-88e685-hnxx7" podStartSLOduration=2.247066082 podStartE2EDuration="2.247066082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:50:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:50:11.246622939 +0000 UTC m=+3588.850930493" watchObservedRunningTime="2026-10-30 22:50:11.247066082 +0000 UTC m=+3588.851373664"
Oct 30 22:50:12.142565 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:50:12.142531    5678 generic.go:356] "Generic (PLEG): container finished" podID="6599cc7c-79c0-4aab-9f78-206e2b64db4b" containerID="c9c115e52235ba0d30e7ae799869f905bc23ac4219f90abe852cb8cc8bae18be" exitCode=0
Oct 30 22:50:12.143125 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:50:12.142584    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-88e685-hnxx7" event={"ID":"6599cc7c-79c0-4aab-9f78-206e2b64db4b","Type":"ContainerDied","Data":"c9c115e52235ba0d30e7ae799869f905bc23ac4219f90abe852cb8cc8bae18be"}
Oct 30 22:50:13.279232 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:50:13.279209    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-88e685-hnxx7"
Oct 30 22:50:13.366278 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:50:13.366243    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6599cc7c-79c0-4aab-9f78-206e2b64db4b-kube-api-access-4ghq7\" (UniqueName: \"kubernetes.io/projected/6599cc7c-79c0-4aab-9f78-206e2b64db4b-kube-api-access-4ghq7\") pod \"6599cc7c-79c0-4aab-9f78-206e2b64db4b\" (UID: \"6599cc7c-79c0-4aab-9f78-206e2b64db4b\") "
Oct 30 22:50:13.366507 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:50:13.366292    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6599cc7c-79c0-4aab-9f78-206e2b64db4b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6599cc7c-79c0-4aab-9f78-206e2b64db4b-maas-api-xks-ca\") pod \"6599cc7c-79c0-4aab-9f78-206e2b64db4b\" (UID: \"6599cc7c-79c0-4aab-9f78-206e2b64db4b\") "
Oct 30 22:50:13.366507 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:50:13.366382    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6599cc7c-79c0-4aab-9f78-206e2b64db4b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6599cc7c-79c0-4aab-9f78-206e2b64db4b-maas-api-openshift-ca\") pod \"6599cc7c-79c0-4aab-9f78-206e2b64db4b\" (UID: \"6599cc7c-79c0-4aab-9f78-206e2b64db4b\") "
Oct 30 22:50:13.366734 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:50:13.366705    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6599cc7c-79c0-4aab-9f78-206e2b64db4b-maas-api-openshift-ca" pod "6599cc7c-79c0-4aab-9f78-206e2b64db4b" (UID: "6599cc7c-79c0-4aab-9f78-206e2b64db4b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:50:13.368388 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:50:13.368367    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6599cc7c-79c0-4aab-9f78-206e2b64db4b-maas-api-xks-ca" pod "6599cc7c-79c0-4aab-9f78-206e2b64db4b" (UID: "6599cc7c-79c0-4aab-9f78-206e2b64db4b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:50:13.368507 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:50:13.368459    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6599cc7c-79c0-4aab-9f78-206e2b64db4b-kube-api-access-4ghq7" pod "6599cc7c-79c0-4aab-9f78-206e2b64db4b" (UID: "6599cc7c-79c0-4aab-9f78-206e2b64db4b"). InnerVolumeSpecName "kube-api-access-4ghq7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:50:13.467432 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:50:13.467324    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6599cc7c-79c0-4aab-9f78-206e2b64db4b-maas-api-openshift-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:50:13.467432 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:50:13.467354    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ghq7\" (UniqueName: \"kubernetes.io/projected/6599cc7c-79c0-4aab-9f78-206e2b64db4b-kube-api-access-4ghq7\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:50:13.467432 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:50:13.467368    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6599cc7c-79c0-4aab-9f78-206e2b64db4b-maas-api-xks-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:50:14.154099 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:50:14.154071    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-88e685-hnxx7"
Oct 30 22:50:14.154099 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:50:14.154077    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-88e685-hnxx7" event={"ID":"6599cc7c-79c0-4aab-9f78-206e2b64db4b","Type":"ContainerDied","Data":"1c70f208578e9da9c2146fb79748b83918b7beb9b7de3a9881a2ac2315fdcf11"}
Oct 30 22:50:14.154318 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:50:14.154131    5678 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1c70f208578e9da9c2146fb79748b83918b7beb9b7de3a9881a2ac2315fdcf11"
Oct 30 22:50:24.492188 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:50:24.492156    5678 scope.go:122] "RemoveContainer" containerID="9e59d1b00f623a152792b538676049c571f446fced47647487639952bf8d35a7"
Oct 30 22:50:24.500950 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:50:24.500929    5678 scope.go:122] "RemoveContainer" containerID="09b39c5bbe65326715ff88f78a1310c1a0d26f19bdee1e101824532beb1d40c2"
Oct 30 22:50:24.508950 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:50:24.508932    5678 scope.go:122] "RemoveContainer" containerID="934e0f7a01465f66c6d9c94b7274d0174490086b5a8396b3c99f915971de1f65"
Oct 30 22:50:27.269248 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:50:27.269212    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-88e685-hnxx7"]
Oct 30 22:50:27.316349 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:50:27.316318    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-88e685-hnxx7"]
Oct 30 22:50:28.521622 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:50:28.521579    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6599cc7c-79c0-4aab-9f78-206e2b64db4b" path="/var/lib/kubelet/pods/6599cc7c-79c0-4aab-9f78-206e2b64db4b/volumes"
Oct 30 22:50:35.717321 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:50:35.717241    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fxhkk"]
Oct 30 22:50:35.764357 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:50:35.764326    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fxhkk"]
Oct 30 22:50:35.846000 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:50:35.845968    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ksd4n"]
Oct 30 22:50:35.846358 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:50:35.846346    5678 cpu_manager.go:414] "containerMap: removing container" podUID="6599cc7c-79c0-4aab-9f78-206e2b64db4b" containerName="revoke-keys"
Oct 30 22:50:35.846424 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:50:35.846359    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6599cc7c-79c0-4aab-9f78-206e2b64db4b" containerName="revoke-keys"
Oct 30 22:50:35.846470 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:50:35.846430    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="6599cc7c-79c0-4aab-9f78-206e2b64db4b" containerName="revoke-keys"
Oct 30 22:50:35.850935 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:50:35.850913    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ksd4n"
Oct 30 22:50:35.871655 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:50:35.871627    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81681875-a3fb-4fc7-b253-940c2370c6cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-ksd4n\" (UID: \"81681875-a3fb-4fc7-b253-940c2370c6cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ksd4n"
Oct 30 22:50:35.871777 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:50:35.871674    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81681875-a3fb-4fc7-b253-940c2370c6cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-ksd4n\" (UID: \"81681875-a3fb-4fc7-b253-940c2370c6cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ksd4n"
Oct 30 22:50:35.871840 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:50:35.871798    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txrzr\" (UniqueName: \"kubernetes.io/projected/81681875-a3fb-4fc7-b253-940c2370c6cb-kube-api-access-txrzr\") pod \"maas-api-revoke-sub-3d4156f1-ksd4n\" (UID: \"81681875-a3fb-4fc7-b253-940c2370c6cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ksd4n"
Oct 30 22:50:35.902497 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:50:35.902469    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5rvcj\""
Oct 30 22:50:35.902639 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:50:35.902469    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:50:35.902639 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:50:35.902526    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:50:35.902639 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:50:35.902526    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:50:35.904982 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:50:35.904950    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ksd4n"]
Oct 30 22:50:35.973270 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:50:35.973184    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txrzr\" (UniqueName: \"kubernetes.io/projected/81681875-a3fb-4fc7-b253-940c2370c6cb-kube-api-access-txrzr\") pod \"maas-api-revoke-sub-3d4156f1-ksd4n\" (UID: \"81681875-a3fb-4fc7-b253-940c2370c6cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ksd4n"
Oct 30 22:50:35.973270 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:50:35.973231    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81681875-a3fb-4fc7-b253-940c2370c6cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-ksd4n\" (UID: \"81681875-a3fb-4fc7-b253-940c2370c6cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ksd4n"
Oct 30 22:50:35.973270 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:50:35.973269    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81681875-a3fb-4fc7-b253-940c2370c6cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-ksd4n\" (UID: \"81681875-a3fb-4fc7-b253-940c2370c6cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ksd4n"
Oct 30 22:50:35.974040 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:50:35.974020    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81681875-a3fb-4fc7-b253-940c2370c6cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-ksd4n\" (UID: \"81681875-a3fb-4fc7-b253-940c2370c6cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ksd4n"
Oct 30 22:50:35.975675 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:50:35.975654    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81681875-a3fb-4fc7-b253-940c2370c6cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-ksd4n\" (UID: \"81681875-a3fb-4fc7-b253-940c2370c6cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ksd4n"
Oct 30 22:50:36.032171 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:50:36.032145    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txrzr\" (UniqueName: \"kubernetes.io/projected/81681875-a3fb-4fc7-b253-940c2370c6cb-kube-api-access-txrzr\") pod \"maas-api-revoke-sub-3d4156f1-ksd4n\" (UID: \"81681875-a3fb-4fc7-b253-940c2370c6cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ksd4n"
Oct 30 22:50:36.162995 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:50:36.162961    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ksd4n"
Oct 30 22:50:36.522941 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:50:36.522903    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa4525e5-59d9-413f-9f38-be7b1f37a359" path="/var/lib/kubelet/pods/fa4525e5-59d9-413f-9f38-be7b1f37a359/volumes"
Oct 30 22:50:36.548014 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:50:36.547985    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ksd4n"]
Oct 30 22:50:36.554882 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:50:36.554838    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81681875_a3fb_4fc7_b253_940c2370c6cb.slice/crio-05d6b8996b05bf9dc32e5846131f6a0d6f06805a0b22677871ad8c69100f36ad WatchSource:0}: Error finding container 05d6b8996b05bf9dc32e5846131f6a0d6f06805a0b22677871ad8c69100f36ad: Status 404 returned error can't find the container with id 05d6b8996b05bf9dc32e5846131f6a0d6f06805a0b22677871ad8c69100f36ad
Oct 30 22:50:37.290877 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:50:37.290844    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ksd4n" event={"ID":"81681875-a3fb-4fc7-b253-940c2370c6cb","Type":"ContainerStarted","Data":"29537a03414716df2ec9d6e9a1e03871cf5d1f40e5a02c3dff8e8c32f441b16b"}
Oct 30 22:50:37.291270 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:50:37.290895    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ksd4n" event={"ID":"81681875-a3fb-4fc7-b253-940c2370c6cb","Type":"ContainerStarted","Data":"05d6b8996b05bf9dc32e5846131f6a0d6f06805a0b22677871ad8c69100f36ad"}
Oct 30 22:50:37.401907 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:50:37.401859    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ksd4n" podStartSLOduration=2.401844626 podStartE2EDuration="2.401844626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:50:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:50:37.400922174 +0000 UTC m=+3615.005229728" watchObservedRunningTime="2026-10-30 22:50:37.401844626 +0000 UTC m=+3615.006152157"
Oct 30 22:50:38.298003 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:50:38.297973    5678 generic.go:356] "Generic (PLEG): container finished" podID="81681875-a3fb-4fc7-b253-940c2370c6cb" containerID="29537a03414716df2ec9d6e9a1e03871cf5d1f40e5a02c3dff8e8c32f441b16b" exitCode=0
Oct 30 22:50:38.298429 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:50:38.298048    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ksd4n" event={"ID":"81681875-a3fb-4fc7-b253-940c2370c6cb","Type":"ContainerDied","Data":"29537a03414716df2ec9d6e9a1e03871cf5d1f40e5a02c3dff8e8c32f441b16b"}
Oct 30 22:50:39.429612 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:50:39.429591    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ksd4n"
Oct 30 22:50:39.508243 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:50:39.508200    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/81681875-a3fb-4fc7-b253-940c2370c6cb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81681875-a3fb-4fc7-b253-940c2370c6cb-maas-api-xks-ca\") pod \"81681875-a3fb-4fc7-b253-940c2370c6cb\" (UID: \"81681875-a3fb-4fc7-b253-940c2370c6cb\") "
Oct 30 22:50:39.508416 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:50:39.508353    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81681875-a3fb-4fc7-b253-940c2370c6cb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81681875-a3fb-4fc7-b253-940c2370c6cb-maas-api-openshift-ca\") pod \"81681875-a3fb-4fc7-b253-940c2370c6cb\" (UID: \"81681875-a3fb-4fc7-b253-940c2370c6cb\") "
Oct 30 22:50:39.508416 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:50:39.508385    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81681875-a3fb-4fc7-b253-940c2370c6cb-kube-api-access-txrzr\" (UniqueName: \"kubernetes.io/projected/81681875-a3fb-4fc7-b253-940c2370c6cb-kube-api-access-txrzr\") pod \"81681875-a3fb-4fc7-b253-940c2370c6cb\" (UID: \"81681875-a3fb-4fc7-b253-940c2370c6cb\") "
Oct 30 22:50:39.508749 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:50:39.508710    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81681875-a3fb-4fc7-b253-940c2370c6cb-maas-api-openshift-ca" pod "81681875-a3fb-4fc7-b253-940c2370c6cb" (UID: "81681875-a3fb-4fc7-b253-940c2370c6cb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:50:39.510108 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:50:39.510084    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81681875-a3fb-4fc7-b253-940c2370c6cb-maas-api-xks-ca" pod "81681875-a3fb-4fc7-b253-940c2370c6cb" (UID: "81681875-a3fb-4fc7-b253-940c2370c6cb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:50:39.510411 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:50:39.510379    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81681875-a3fb-4fc7-b253-940c2370c6cb-kube-api-access-txrzr" pod "81681875-a3fb-4fc7-b253-940c2370c6cb" (UID: "81681875-a3fb-4fc7-b253-940c2370c6cb"). InnerVolumeSpecName "kube-api-access-txrzr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:50:39.609505 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:50:39.609418    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81681875-a3fb-4fc7-b253-940c2370c6cb-maas-api-openshift-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:50:39.609505 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:50:39.609453    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txrzr\" (UniqueName: \"kubernetes.io/projected/81681875-a3fb-4fc7-b253-940c2370c6cb-kube-api-access-txrzr\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:50:39.609505 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:50:39.609467    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81681875-a3fb-4fc7-b253-940c2370c6cb-maas-api-xks-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:50:40.315664 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:50:40.315635    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ksd4n"
Oct 30 22:50:40.315873 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:50:40.315632    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ksd4n" event={"ID":"81681875-a3fb-4fc7-b253-940c2370c6cb","Type":"ContainerDied","Data":"05d6b8996b05bf9dc32e5846131f6a0d6f06805a0b22677871ad8c69100f36ad"}
Oct 30 22:50:40.315873 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:50:40.315768    5678 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="05d6b8996b05bf9dc32e5846131f6a0d6f06805a0b22677871ad8c69100f36ad"
Oct 30 22:50:40.571127 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:50:40.571050    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 22:50:48.046495 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:50:48.046458    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-hv6cz"]
Oct 30 22:50:48.093609 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:50:48.093576    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-hv6cz"]
Oct 30 22:50:48.521819 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:50:48.521789    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62a9a748-fd94-4177-be46-c2cd40a08b98" path="/var/lib/kubelet/pods/62a9a748-fd94-4177-be46-c2cd40a08b98/volumes"
Oct 30 22:51:10.571715 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:51:10.571678    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 22:51:12.049818 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:51:12.049785    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-8zfl6"]
Oct 30 22:51:12.096661 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:51:12.096627    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-8zfl6"]
Oct 30 22:51:12.521477 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:51:12.521446    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c6e09de-bf26-4a43-a8fc-5a2bdf2ec10c" path="/var/lib/kubelet/pods/1c6e09de-bf26-4a43-a8fc-5a2bdf2ec10c/volumes"
Oct 30 22:51:24.536195 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:51:24.536165    5678 scope.go:122] "RemoveContainer" containerID="a25a9f52dcf77517d2278ec6cedf32bfe060cd14547d55cd90fad492a773b14c"
Oct 30 22:51:40.570413 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:51:40.570361    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 22:52:10.571509 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:52:10.571429    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 22:52:24.559975 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:52:24.559937    5678 scope.go:122] "RemoveContainer" containerID="13f8b4a28e7777f84945526541938e6b660e6a8ca3f42f94beb4a7551ebe5a32"
Oct 30 22:52:24.569214 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:52:24.569191    5678 scope.go:122] "RemoveContainer" containerID="6fca4c5799b297bee1627f65e3808b5ee801425223cc7f33a1482b5b984827b0"
Oct 30 22:52:24.577148 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:52:24.577130    5678 scope.go:122] "RemoveContainer" containerID="61a059d9b6b7117c4927686b0a2614357debe5e372ab66a6d9ec3eb7a63199d1"
Oct 30 22:52:40.573105 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:52:40.573064    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 22:52:45.047959 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:52:45.047915    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-nz85w"]
Oct 30 22:52:45.094102 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:52:45.094031    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-nz85w"]
Oct 30 22:52:45.828840 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:52:45.828806    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-h9jv7"]
Oct 30 22:52:45.829194 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:52:45.829181    5678 cpu_manager.go:414] "containerMap: removing container" podUID="81681875-a3fb-4fc7-b253-940c2370c6cb" containerName="revoke-keys"
Oct 30 22:52:45.829241 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:52:45.829196    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81681875-a3fb-4fc7-b253-940c2370c6cb" containerName="revoke-keys"
Oct 30 22:52:45.829279 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:52:45.829269    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="81681875-a3fb-4fc7-b253-940c2370c6cb" containerName="revoke-keys"
Oct 30 22:52:45.832379 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:52:45.832355    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-h9jv7"
Oct 30 22:52:45.920879 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:52:45.920846    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nw2pl\" (UniqueName: \"kubernetes.io/projected/331b8c93-8ca7-4cda-8364-e146dda3aa33-kube-api-access-nw2pl\") pod \"maas-api-revoke-sub-d576e68b-h9jv7\" (UID: \"331b8c93-8ca7-4cda-8364-e146dda3aa33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-h9jv7"
Oct 30 22:52:45.920879 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:52:45.920889    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/331b8c93-8ca7-4cda-8364-e146dda3aa33-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-h9jv7\" (UID: \"331b8c93-8ca7-4cda-8364-e146dda3aa33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-h9jv7"
Oct 30 22:52:45.921081 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:52:45.920904    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/331b8c93-8ca7-4cda-8364-e146dda3aa33-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-h9jv7\" (UID: \"331b8c93-8ca7-4cda-8364-e146dda3aa33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-h9jv7"
Oct 30 22:52:45.932194 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:52:45.932168    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:52:45.932355 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:52:45.932196    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:52:45.932355 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:52:45.932205    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:52:45.932355 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:52:45.932273    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5rvcj\""
Oct 30 22:52:45.934051 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:52:45.934032    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-h9jv7"]
Oct 30 22:52:46.021997 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:52:46.021972    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nw2pl\" (UniqueName: \"kubernetes.io/projected/331b8c93-8ca7-4cda-8364-e146dda3aa33-kube-api-access-nw2pl\") pod \"maas-api-revoke-sub-d576e68b-h9jv7\" (UID: \"331b8c93-8ca7-4cda-8364-e146dda3aa33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-h9jv7"
Oct 30 22:52:46.022176 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:52:46.022005    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/331b8c93-8ca7-4cda-8364-e146dda3aa33-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-h9jv7\" (UID: \"331b8c93-8ca7-4cda-8364-e146dda3aa33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-h9jv7"
Oct 30 22:52:46.022176 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:52:46.022025    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/331b8c93-8ca7-4cda-8364-e146dda3aa33-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-h9jv7\" (UID: \"331b8c93-8ca7-4cda-8364-e146dda3aa33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-h9jv7"
Oct 30 22:52:46.022772 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:52:46.022754    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/331b8c93-8ca7-4cda-8364-e146dda3aa33-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-h9jv7\" (UID: \"331b8c93-8ca7-4cda-8364-e146dda3aa33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-h9jv7"
Oct 30 22:52:46.024362 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:52:46.024335    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/331b8c93-8ca7-4cda-8364-e146dda3aa33-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-h9jv7\" (UID: \"331b8c93-8ca7-4cda-8364-e146dda3aa33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-h9jv7"
Oct 30 22:52:46.080830 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:52:46.080765    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nw2pl\" (UniqueName: \"kubernetes.io/projected/331b8c93-8ca7-4cda-8364-e146dda3aa33-kube-api-access-nw2pl\") pod \"maas-api-revoke-sub-d576e68b-h9jv7\" (UID: \"331b8c93-8ca7-4cda-8364-e146dda3aa33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-h9jv7"
Oct 30 22:52:46.145747 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:52:46.145723    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-h9jv7"
Oct 30 22:52:46.523084 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:52:46.523048    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2ff7913-4053-42e7-8b66-49ad5b4ab7ba" path="/var/lib/kubelet/pods/a2ff7913-4053-42e7-8b66-49ad5b4ab7ba/volumes"
Oct 30 22:52:46.524549 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:52:46.524525    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-h9jv7"]
Oct 30 22:52:46.531540 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:52:46.531515    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod331b8c93_8ca7_4cda_8364_e146dda3aa33.slice/crio-fc5d0694fce74a77df70c37c2cca6208b5aa113d3a51826f7225c4404623b618 WatchSource:0}: Error finding container fc5d0694fce74a77df70c37c2cca6208b5aa113d3a51826f7225c4404623b618: Status 404 returned error can't find the container with id fc5d0694fce74a77df70c37c2cca6208b5aa113d3a51826f7225c4404623b618
Oct 30 22:52:47.015100 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:52:47.015063    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-h9jv7" event={"ID":"331b8c93-8ca7-4cda-8364-e146dda3aa33","Type":"ContainerStarted","Data":"c4bec6f57816c7e62c8a8093244b6427c0c83a9a228f0030eda1b50a7ff08675"}
Oct 30 22:52:47.015282 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:52:47.015125    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-h9jv7" event={"ID":"331b8c93-8ca7-4cda-8364-e146dda3aa33","Type":"ContainerStarted","Data":"fc5d0694fce74a77df70c37c2cca6208b5aa113d3a51826f7225c4404623b618"}
Oct 30 22:52:47.150576 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:52:47.150527    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-h9jv7" podStartSLOduration=2.150514147 podStartE2EDuration="2.150514147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:52:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:52:47.150172261 +0000 UTC m=+3744.754479813" watchObservedRunningTime="2026-10-30 22:52:47.150514147 +0000 UTC m=+3744.754821757"
Oct 30 22:52:48.021156 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:52:48.021120    5678 generic.go:356] "Generic (PLEG): container finished" podID="331b8c93-8ca7-4cda-8364-e146dda3aa33" containerID="c4bec6f57816c7e62c8a8093244b6427c0c83a9a228f0030eda1b50a7ff08675" exitCode=0
Oct 30 22:52:48.021328 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:52:48.021201    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-h9jv7" event={"ID":"331b8c93-8ca7-4cda-8364-e146dda3aa33","Type":"ContainerDied","Data":"c4bec6f57816c7e62c8a8093244b6427c0c83a9a228f0030eda1b50a7ff08675"}
Oct 30 22:52:49.155200 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:52:49.155171    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-h9jv7"
Oct 30 22:52:49.253924 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:52:49.253892    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/331b8c93-8ca7-4cda-8364-e146dda3aa33-kube-api-access-nw2pl\" (UniqueName: \"kubernetes.io/projected/331b8c93-8ca7-4cda-8364-e146dda3aa33-kube-api-access-nw2pl\") pod \"331b8c93-8ca7-4cda-8364-e146dda3aa33\" (UID: \"331b8c93-8ca7-4cda-8364-e146dda3aa33\") "
Oct 30 22:52:49.254093 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:52:49.253948    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/331b8c93-8ca7-4cda-8364-e146dda3aa33-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/331b8c93-8ca7-4cda-8364-e146dda3aa33-maas-api-openshift-ca\") pod \"331b8c93-8ca7-4cda-8364-e146dda3aa33\" (UID: \"331b8c93-8ca7-4cda-8364-e146dda3aa33\") "
Oct 30 22:52:49.254093 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:52:49.254018    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/331b8c93-8ca7-4cda-8364-e146dda3aa33-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/331b8c93-8ca7-4cda-8364-e146dda3aa33-maas-api-xks-ca\") pod \"331b8c93-8ca7-4cda-8364-e146dda3aa33\" (UID: \"331b8c93-8ca7-4cda-8364-e146dda3aa33\") "
Oct 30 22:52:49.254304 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:52:49.254280    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/331b8c93-8ca7-4cda-8364-e146dda3aa33-maas-api-openshift-ca" pod "331b8c93-8ca7-4cda-8364-e146dda3aa33" (UID: "331b8c93-8ca7-4cda-8364-e146dda3aa33"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:52:49.255934 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:52:49.255916    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/331b8c93-8ca7-4cda-8364-e146dda3aa33-kube-api-access-nw2pl" pod "331b8c93-8ca7-4cda-8364-e146dda3aa33" (UID: "331b8c93-8ca7-4cda-8364-e146dda3aa33"). InnerVolumeSpecName "kube-api-access-nw2pl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:52:49.255995 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:52:49.255979    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/331b8c93-8ca7-4cda-8364-e146dda3aa33-maas-api-xks-ca" pod "331b8c93-8ca7-4cda-8364-e146dda3aa33" (UID: "331b8c93-8ca7-4cda-8364-e146dda3aa33"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:52:49.354768 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:52:49.354691    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/331b8c93-8ca7-4cda-8364-e146dda3aa33-maas-api-xks-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:52:49.354768 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:52:49.354721    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nw2pl\" (UniqueName: \"kubernetes.io/projected/331b8c93-8ca7-4cda-8364-e146dda3aa33-kube-api-access-nw2pl\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:52:49.354768 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:52:49.354733    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/331b8c93-8ca7-4cda-8364-e146dda3aa33-maas-api-openshift-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:52:50.034394 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:52:50.034364    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-h9jv7"
Oct 30 22:52:50.034583 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:52:50.034393    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-h9jv7" event={"ID":"331b8c93-8ca7-4cda-8364-e146dda3aa33","Type":"ContainerDied","Data":"fc5d0694fce74a77df70c37c2cca6208b5aa113d3a51826f7225c4404623b618"}
Oct 30 22:52:50.034583 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:52:50.034473    5678 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fc5d0694fce74a77df70c37c2cca6208b5aa113d3a51826f7225c4404623b618"
Oct 30 22:52:56.054952 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:52:56.054915    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-rmspf"]
Oct 30 22:52:56.093108 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:52:56.093074    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-rmspf"]
Oct 30 22:52:56.521206 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:52:56.521170    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ce8b029-1e6a-42e0-a0af-37e067c7a618" path="/var/lib/kubelet/pods/5ce8b029-1e6a-42e0-a0af-37e067c7a618/volumes"
Oct 30 22:53:10.571488 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:53:10.571449    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 22:53:15.006231 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:53:15.006193    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-pmz6j"]
Oct 30 22:53:15.006695 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:53:15.006582    5678 cpu_manager.go:414] "containerMap: removing container" podUID="331b8c93-8ca7-4cda-8364-e146dda3aa33" containerName="revoke-keys"
Oct 30 22:53:15.006695 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:53:15.006593    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="331b8c93-8ca7-4cda-8364-e146dda3aa33" containerName="revoke-keys"
Oct 30 22:53:15.006695 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:53:15.006676    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="331b8c93-8ca7-4cda-8364-e146dda3aa33" containerName="revoke-keys"
Oct 30 22:53:15.009868 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:53:15.009849    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-pmz6j"
Oct 30 22:53:15.059673 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:53:15.059646    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:53:15.059931 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:53:15.059909    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5rvcj\""
Oct 30 22:53:15.060034 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:53:15.059995    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:53:15.094583 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:53:15.094548    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ded5b225-d97d-4bff-be9a-dd752ddb1edf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-pmz6j\" (UID: \"ded5b225-d97d-4bff-be9a-dd752ddb1edf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-pmz6j"
Oct 30 22:53:15.094721 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:53:15.094598    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qv9pz\" (UniqueName: \"kubernetes.io/projected/ded5b225-d97d-4bff-be9a-dd752ddb1edf-kube-api-access-qv9pz\") pod \"maas-api-revoke-sub-d2bc93bc-pmz6j\" (UID: \"ded5b225-d97d-4bff-be9a-dd752ddb1edf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-pmz6j"
Oct 30 22:53:15.094816 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:53:15.094788    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ded5b225-d97d-4bff-be9a-dd752ddb1edf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-pmz6j\" (UID: \"ded5b225-d97d-4bff-be9a-dd752ddb1edf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-pmz6j"
Oct 30 22:53:15.106995 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:53:15.106958    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:53:15.108854 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:53:15.108835    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-pmz6j"]
Oct 30 22:53:15.195227 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:53:15.195197    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ded5b225-d97d-4bff-be9a-dd752ddb1edf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-pmz6j\" (UID: \"ded5b225-d97d-4bff-be9a-dd752ddb1edf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-pmz6j"
Oct 30 22:53:15.195227 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:53:15.195239    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ded5b225-d97d-4bff-be9a-dd752ddb1edf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-pmz6j\" (UID: \"ded5b225-d97d-4bff-be9a-dd752ddb1edf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-pmz6j"
Oct 30 22:53:15.195464 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:53:15.195270    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qv9pz\" (UniqueName: \"kubernetes.io/projected/ded5b225-d97d-4bff-be9a-dd752ddb1edf-kube-api-access-qv9pz\") pod \"maas-api-revoke-sub-d2bc93bc-pmz6j\" (UID: \"ded5b225-d97d-4bff-be9a-dd752ddb1edf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-pmz6j"
Oct 30 22:53:15.196165 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:53:15.196138    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ded5b225-d97d-4bff-be9a-dd752ddb1edf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-pmz6j\" (UID: \"ded5b225-d97d-4bff-be9a-dd752ddb1edf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-pmz6j"
Oct 30 22:53:15.197745 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:53:15.197723    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ded5b225-d97d-4bff-be9a-dd752ddb1edf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-pmz6j\" (UID: \"ded5b225-d97d-4bff-be9a-dd752ddb1edf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-pmz6j"
Oct 30 22:53:15.254588 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:53:15.254556    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qv9pz\" (UniqueName: \"kubernetes.io/projected/ded5b225-d97d-4bff-be9a-dd752ddb1edf-kube-api-access-qv9pz\") pod \"maas-api-revoke-sub-d2bc93bc-pmz6j\" (UID: \"ded5b225-d97d-4bff-be9a-dd752ddb1edf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-pmz6j"
Oct 30 22:53:15.322546 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:53:15.322471    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-pmz6j"
Oct 30 22:53:15.708495 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:53:15.708459    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-pmz6j"]
Oct 30 22:53:15.715571 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:53:15.715542    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podded5b225_d97d_4bff_be9a_dd752ddb1edf.slice/crio-8e106a204fed3929671dc69ab3b4fefc3131b8d3677636d3079fab8b9ee8399d WatchSource:0}: Error finding container 8e106a204fed3929671dc69ab3b4fefc3131b8d3677636d3079fab8b9ee8399d: Status 404 returned error can't find the container with id 8e106a204fed3929671dc69ab3b4fefc3131b8d3677636d3079fab8b9ee8399d
Oct 30 22:53:16.176999 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:53:16.176964    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-pmz6j" event={"ID":"ded5b225-d97d-4bff-be9a-dd752ddb1edf","Type":"ContainerStarted","Data":"14c84c7fe4662789b34bc6c869214ff6d76b57497b7e243e3964af905578dba1"}
Oct 30 22:53:16.177463 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:53:16.177018    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-pmz6j" event={"ID":"ded5b225-d97d-4bff-be9a-dd752ddb1edf","Type":"ContainerStarted","Data":"8e106a204fed3929671dc69ab3b4fefc3131b8d3677636d3079fab8b9ee8399d"}
Oct 30 22:53:16.287741 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:53:16.287644    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-pmz6j" podStartSLOduration=2.287628998 podStartE2EDuration="2.287628998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:53:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:53:16.286233948 +0000 UTC m=+3773.890541503" watchObservedRunningTime="2026-10-30 22:53:16.287628998 +0000 UTC m=+3773.891936586"
Oct 30 22:53:17.183787 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:53:17.183753    5678 generic.go:356] "Generic (PLEG): container finished" podID="ded5b225-d97d-4bff-be9a-dd752ddb1edf" containerID="14c84c7fe4662789b34bc6c869214ff6d76b57497b7e243e3964af905578dba1" exitCode=0
Oct 30 22:53:17.184178 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:53:17.183818    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-pmz6j" event={"ID":"ded5b225-d97d-4bff-be9a-dd752ddb1edf","Type":"ContainerDied","Data":"14c84c7fe4662789b34bc6c869214ff6d76b57497b7e243e3964af905578dba1"}
Oct 30 22:53:18.319732 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:53:18.319706    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-pmz6j"
Oct 30 22:53:18.425929 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:53:18.425892    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ded5b225-d97d-4bff-be9a-dd752ddb1edf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ded5b225-d97d-4bff-be9a-dd752ddb1edf-maas-api-xks-ca\") pod \"ded5b225-d97d-4bff-be9a-dd752ddb1edf\" (UID: \"ded5b225-d97d-4bff-be9a-dd752ddb1edf\") "
Oct 30 22:53:18.426113 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:53:18.425976    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ded5b225-d97d-4bff-be9a-dd752ddb1edf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ded5b225-d97d-4bff-be9a-dd752ddb1edf-maas-api-openshift-ca\") pod \"ded5b225-d97d-4bff-be9a-dd752ddb1edf\" (UID: \"ded5b225-d97d-4bff-be9a-dd752ddb1edf\") "
Oct 30 22:53:18.426113 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:53:18.425996    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ded5b225-d97d-4bff-be9a-dd752ddb1edf-kube-api-access-qv9pz\" (UniqueName: \"kubernetes.io/projected/ded5b225-d97d-4bff-be9a-dd752ddb1edf-kube-api-access-qv9pz\") pod \"ded5b225-d97d-4bff-be9a-dd752ddb1edf\" (UID: \"ded5b225-d97d-4bff-be9a-dd752ddb1edf\") "
Oct 30 22:53:18.426359 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:53:18.426330    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ded5b225-d97d-4bff-be9a-dd752ddb1edf-maas-api-openshift-ca" pod "ded5b225-d97d-4bff-be9a-dd752ddb1edf" (UID: "ded5b225-d97d-4bff-be9a-dd752ddb1edf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:53:18.428088 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:53:18.428062    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ded5b225-d97d-4bff-be9a-dd752ddb1edf-maas-api-xks-ca" pod "ded5b225-d97d-4bff-be9a-dd752ddb1edf" (UID: "ded5b225-d97d-4bff-be9a-dd752ddb1edf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:53:18.428190 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:53:18.428115    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ded5b225-d97d-4bff-be9a-dd752ddb1edf-kube-api-access-qv9pz" pod "ded5b225-d97d-4bff-be9a-dd752ddb1edf" (UID: "ded5b225-d97d-4bff-be9a-dd752ddb1edf"). InnerVolumeSpecName "kube-api-access-qv9pz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:53:18.527707 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:53:18.527640    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ded5b225-d97d-4bff-be9a-dd752ddb1edf-maas-api-xks-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:53:18.527707 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:53:18.527663    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ded5b225-d97d-4bff-be9a-dd752ddb1edf-maas-api-openshift-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:53:18.527707 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:53:18.527675    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qv9pz\" (UniqueName: \"kubernetes.io/projected/ded5b225-d97d-4bff-be9a-dd752ddb1edf-kube-api-access-qv9pz\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:53:19.195108 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:53:19.195076    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-pmz6j" event={"ID":"ded5b225-d97d-4bff-be9a-dd752ddb1edf","Type":"ContainerDied","Data":"8e106a204fed3929671dc69ab3b4fefc3131b8d3677636d3079fab8b9ee8399d"}
Oct 30 22:53:19.195281 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:53:19.195111    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-pmz6j"
Oct 30 22:53:19.195281 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:53:19.195128    5678 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8e106a204fed3929671dc69ab3b4fefc3131b8d3677636d3079fab8b9ee8399d"
Oct 30 22:53:24.603381 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:53:24.603349    5678 scope.go:122] "RemoveContainer" containerID="118df99c26939f10d7f38d8a632d850632d915b5056574c3bcd24ee55becc8b9"
Oct 30 22:53:39.316809 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:53:39.316771    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4334bf-5678f547b-cvfkg"]
Oct 30 22:53:39.317349 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:53:39.317328    5678 cpu_manager.go:414] "containerMap: removing container" podUID="ded5b225-d97d-4bff-be9a-dd752ddb1edf" containerName="revoke-keys"
Oct 30 22:53:39.317470 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:53:39.317350    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ded5b225-d97d-4bff-be9a-dd752ddb1edf" containerName="revoke-keys"
Oct 30 22:53:39.317540 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:53:39.317471    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="ded5b225-d97d-4bff-be9a-dd752ddb1edf" containerName="revoke-keys"
Oct 30 22:53:39.320665 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:53:39.320641    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4334bf-5678f547b-cvfkg"
Oct 30 22:53:39.417607 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:53:39.417570    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-4334bf-serving-cert\""
Oct 30 22:53:39.417607 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:53:39.417581    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:53:39.417833 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:53:39.417582    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:53:39.417833 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:53:39.417669    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 30 22:53:39.417833 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:53:39.417760    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-jglqk\""
Oct 30 22:53:39.419832 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:53:39.419815    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4334bf-5678f547b-cvfkg"]
Oct 30 22:53:39.424371 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:53:39.424349    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/abb9d44c-4965-46de-ad45-3d15b8a09630-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-4334bf-5678f547b-cvfkg\" (UID: \"abb9d44c-4965-46de-ad45-3d15b8a09630\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4334bf-5678f547b-cvfkg"
Oct 30 22:53:39.424522 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:53:39.424439    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abb9d44c-4965-46de-ad45-3d15b8a09630-metrics-certs\") pod \"maas-api-e2e-worker-single-main-4334bf-5678f547b-cvfkg\" (UID: \"abb9d44c-4965-46de-ad45-3d15b8a09630\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4334bf-5678f547b-cvfkg"
Oct 30 22:53:39.424522 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:53:39.424498    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/abb9d44c-4965-46de-ad45-3d15b8a09630-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-4334bf-5678f547b-cvfkg\" (UID: \"abb9d44c-4965-46de-ad45-3d15b8a09630\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4334bf-5678f547b-cvfkg"
Oct 30 22:53:39.424630 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:53:39.424527    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlqdr\" (UniqueName: \"kubernetes.io/projected/abb9d44c-4965-46de-ad45-3d15b8a09630-kube-api-access-dlqdr\") pod \"maas-api-e2e-worker-single-main-4334bf-5678f547b-cvfkg\" (UID: \"abb9d44c-4965-46de-ad45-3d15b8a09630\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4334bf-5678f547b-cvfkg"
Oct 30 22:53:39.525826 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:53:39.525793    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/abb9d44c-4965-46de-ad45-3d15b8a09630-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-4334bf-5678f547b-cvfkg\" (UID: \"abb9d44c-4965-46de-ad45-3d15b8a09630\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4334bf-5678f547b-cvfkg"
Oct 30 22:53:39.525997 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:53:39.525878    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abb9d44c-4965-46de-ad45-3d15b8a09630-metrics-certs\") pod \"maas-api-e2e-worker-single-main-4334bf-5678f547b-cvfkg\" (UID: \"abb9d44c-4965-46de-ad45-3d15b8a09630\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4334bf-5678f547b-cvfkg"
Oct 30 22:53:39.525997 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:53:39.525910    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/abb9d44c-4965-46de-ad45-3d15b8a09630-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-4334bf-5678f547b-cvfkg\" (UID: \"abb9d44c-4965-46de-ad45-3d15b8a09630\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4334bf-5678f547b-cvfkg"
Oct 30 22:53:39.525997 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:53:39.525937    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlqdr\" (UniqueName: \"kubernetes.io/projected/abb9d44c-4965-46de-ad45-3d15b8a09630-kube-api-access-dlqdr\") pod \"maas-api-e2e-worker-single-main-4334bf-5678f547b-cvfkg\" (UID: \"abb9d44c-4965-46de-ad45-3d15b8a09630\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4334bf-5678f547b-cvfkg"
Oct 30 22:53:39.526126 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:53:39.526031    5678 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4334bf-serving-cert: secret "maas-api-e2e-worker-single-main-4334bf-serving-cert" not found
Oct 30 22:53:39.526126 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:53:39.526107    5678 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/abb9d44c-4965-46de-ad45-3d15b8a09630-maas-api-tls podName:abb9d44c-4965-46de-ad45-3d15b8a09630 nodeName:}" failed. No retries permitted until 2026-10-30 22:53:40.026084355 +0000 UTC m=+3797.630391900 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/abb9d44c-4965-46de-ad45-3d15b8a09630-maas-api-tls") pod "maas-api-e2e-worker-single-main-4334bf-5678f547b-cvfkg" (UID: "abb9d44c-4965-46de-ad45-3d15b8a09630") : secret "maas-api-e2e-worker-single-main-4334bf-serving-cert" not found
Oct 30 22:53:39.526522 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:53:39.526505    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/abb9d44c-4965-46de-ad45-3d15b8a09630-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-4334bf-5678f547b-cvfkg\" (UID: \"abb9d44c-4965-46de-ad45-3d15b8a09630\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4334bf-5678f547b-cvfkg"
Oct 30 22:53:39.528152 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:53:39.528133    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abb9d44c-4965-46de-ad45-3d15b8a09630-metrics-certs\") pod \"maas-api-e2e-worker-single-main-4334bf-5678f547b-cvfkg\" (UID: \"abb9d44c-4965-46de-ad45-3d15b8a09630\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4334bf-5678f547b-cvfkg"
Oct 30 22:53:39.584632 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:53:39.584560    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlqdr\" (UniqueName: \"kubernetes.io/projected/abb9d44c-4965-46de-ad45-3d15b8a09630-kube-api-access-dlqdr\") pod \"maas-api-e2e-worker-single-main-4334bf-5678f547b-cvfkg\" (UID: \"abb9d44c-4965-46de-ad45-3d15b8a09630\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4334bf-5678f547b-cvfkg"
Oct 30 22:53:40.032120 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:53:40.032081    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/abb9d44c-4965-46de-ad45-3d15b8a09630-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-4334bf-5678f547b-cvfkg\" (UID: \"abb9d44c-4965-46de-ad45-3d15b8a09630\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4334bf-5678f547b-cvfkg"
Oct 30 22:53:40.034544 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:53:40.034508    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/abb9d44c-4965-46de-ad45-3d15b8a09630-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-4334bf-5678f547b-cvfkg\" (UID: \"abb9d44c-4965-46de-ad45-3d15b8a09630\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4334bf-5678f547b-cvfkg"
Oct 30 22:53:40.233789 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:53:40.233748    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4334bf-5678f547b-cvfkg"
Oct 30 22:53:40.570315 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:53:40.570228    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 22:53:40.629001 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:53:40.613801    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4334bf-5678f547b-cvfkg"]
Oct 30 22:53:40.629001 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:53:40.615151    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabb9d44c_4965_46de_ad45_3d15b8a09630.slice/crio-af2a594df74707a0a9f10738b672472e4d653b6576c39ba76c382607a3dd1a5f WatchSource:0}: Error finding container af2a594df74707a0a9f10738b672472e4d653b6576c39ba76c382607a3dd1a5f: Status 404 returned error can't find the container with id af2a594df74707a0a9f10738b672472e4d653b6576c39ba76c382607a3dd1a5f
Oct 30 22:53:40.629001 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:53:40.616478    5678 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 22:53:41.308793 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:53:41.308759    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4334bf-5678f547b-cvfkg" event={"ID":"abb9d44c-4965-46de-ad45-3d15b8a09630","Type":"ContainerStarted","Data":"af2a594df74707a0a9f10738b672472e4d653b6576c39ba76c382607a3dd1a5f"}
Oct 30 22:53:42.315432 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:53:42.315366    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4334bf-5678f547b-cvfkg" event={"ID":"abb9d44c-4965-46de-ad45-3d15b8a09630","Type":"ContainerStarted","Data":"dfa7cb30e239e72eb0c901a73d353107b3373821ff4d7e3c5eaa75358f794af4"}
Oct 30 22:53:42.315850 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:53:42.315467    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4334bf-5678f547b-cvfkg"
Oct 30 22:53:42.428322 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:53:42.428270    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4334bf-5678f547b-cvfkg" podStartSLOduration=2.580652673 podStartE2EDuration="3.428256679s" totalImagesPullingTime="847.604006ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:53:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:53:42.427628358 +0000 UTC m=+3800.031935912" watchObservedRunningTime="2026-10-30 22:53:42.428256679 +0000 UTC m=+3800.032564232"
Oct 30 22:53:48.325248 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:53:48.325223    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4334bf-5678f547b-cvfkg"
Oct 30 22:54:10.571361 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:54:10.571321    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 22:54:24.625342 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:54:24.625309    5678 scope.go:122] "RemoveContainer" containerID="149cf47f78cb0cf819b29adad90663f14fc28ea0844f6eda6abe719febd3a3be"
Oct 30 22:54:24.634043 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:54:24.634025    5678 scope.go:122] "RemoveContainer" containerID="3b37b82e8475780de45165bc40d6eb6400a0d0b18dac0bebd4be3048a7071158"
Oct 30 22:54:24.641704 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:54:24.641683    5678 scope.go:122] "RemoveContainer" containerID="1d4c3f291f44c6322db2cb8fa7e8fb11ba0d14eb3641909430d7bffe914003e5"
Oct 30 22:54:40.571514 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:54:40.571474    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 22:55:01.047198 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:55:01.047160    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78019b2d-vqqll"]
Oct 30 22:55:01.094190 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:55:01.094158    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78019b2d-vqqll"]
Oct 30 22:55:02.522219 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:55:02.522186    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d78034ad-6371-41ef-bb44-f002da98a5a5" path="/var/lib/kubelet/pods/d78034ad-6371-41ef-bb44-f002da98a5a5/volumes"
Oct 30 22:55:08.799519 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:55:08.799485    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4334bf-5678f547b-cvfkg"]
Oct 30 22:55:08.799962 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:55:08.799689    5678 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4334bf-5678f547b-cvfkg" podUID="abb9d44c-4965-46de-ad45-3d15b8a09630" containerName="maas-api" containerID="cri-o://dfa7cb30e239e72eb0c901a73d353107b3373821ff4d7e3c5eaa75358f794af4" gracePeriod=30
Oct 30 22:55:10.049062 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:55:10.049036    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4334bf-5678f547b-cvfkg"
Oct 30 22:55:10.157071 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:55:10.156992    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abb9d44c-4965-46de-ad45-3d15b8a09630-kube-api-access-dlqdr\" (UniqueName: \"kubernetes.io/projected/abb9d44c-4965-46de-ad45-3d15b8a09630-kube-api-access-dlqdr\") pod \"abb9d44c-4965-46de-ad45-3d15b8a09630\" (UID: \"abb9d44c-4965-46de-ad45-3d15b8a09630\") "
Oct 30 22:55:10.157071 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:55:10.157035    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abb9d44c-4965-46de-ad45-3d15b8a09630-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/abb9d44c-4965-46de-ad45-3d15b8a09630-openshift-service-ca\") pod \"abb9d44c-4965-46de-ad45-3d15b8a09630\" (UID: \"abb9d44c-4965-46de-ad45-3d15b8a09630\") "
Oct 30 22:55:10.157278 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:55:10.157100    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/abb9d44c-4965-46de-ad45-3d15b8a09630-metrics-certs\" (UniqueName: \"kubernetes.io/secret/abb9d44c-4965-46de-ad45-3d15b8a09630-metrics-certs\") pod \"abb9d44c-4965-46de-ad45-3d15b8a09630\" (UID: \"abb9d44c-4965-46de-ad45-3d15b8a09630\") "
Oct 30 22:55:10.157278 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:55:10.157142    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/abb9d44c-4965-46de-ad45-3d15b8a09630-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/abb9d44c-4965-46de-ad45-3d15b8a09630-maas-api-tls\") pod \"abb9d44c-4965-46de-ad45-3d15b8a09630\" (UID: \"abb9d44c-4965-46de-ad45-3d15b8a09630\") "
Oct 30 22:55:10.157546 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:55:10.157510    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abb9d44c-4965-46de-ad45-3d15b8a09630-openshift-service-ca" pod "abb9d44c-4965-46de-ad45-3d15b8a09630" (UID: "abb9d44c-4965-46de-ad45-3d15b8a09630"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:55:10.159173 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:55:10.159141    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abb9d44c-4965-46de-ad45-3d15b8a09630-maas-api-tls" pod "abb9d44c-4965-46de-ad45-3d15b8a09630" (UID: "abb9d44c-4965-46de-ad45-3d15b8a09630"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:55:10.159265 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:55:10.159181    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abb9d44c-4965-46de-ad45-3d15b8a09630-kube-api-access-dlqdr" pod "abb9d44c-4965-46de-ad45-3d15b8a09630" (UID: "abb9d44c-4965-46de-ad45-3d15b8a09630"). InnerVolumeSpecName "kube-api-access-dlqdr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:55:10.159265 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:55:10.159208    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abb9d44c-4965-46de-ad45-3d15b8a09630-metrics-certs" pod "abb9d44c-4965-46de-ad45-3d15b8a09630" (UID: "abb9d44c-4965-46de-ad45-3d15b8a09630"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:55:10.257968 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:55:10.257936    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlqdr\" (UniqueName: \"kubernetes.io/projected/abb9d44c-4965-46de-ad45-3d15b8a09630-kube-api-access-dlqdr\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:55:10.257968 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:55:10.257961    5678 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/abb9d44c-4965-46de-ad45-3d15b8a09630-openshift-service-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:55:10.257968 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:55:10.257972    5678 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abb9d44c-4965-46de-ad45-3d15b8a09630-metrics-certs\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:55:10.258209 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:55:10.257982    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/abb9d44c-4965-46de-ad45-3d15b8a09630-maas-api-tls\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:55:10.571952 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:55:10.571916    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 22:55:10.774312 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:55:10.774272    5678 generic.go:356] "Generic (PLEG): container finished" podID="abb9d44c-4965-46de-ad45-3d15b8a09630" containerID="dfa7cb30e239e72eb0c901a73d353107b3373821ff4d7e3c5eaa75358f794af4" exitCode=0
Oct 30 22:55:10.774512 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:55:10.774322    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4334bf-5678f547b-cvfkg"
Oct 30 22:55:10.774512 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:55:10.774356    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4334bf-5678f547b-cvfkg" event={"ID":"abb9d44c-4965-46de-ad45-3d15b8a09630","Type":"ContainerDied","Data":"dfa7cb30e239e72eb0c901a73d353107b3373821ff4d7e3c5eaa75358f794af4"}
Oct 30 22:55:10.774512 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:55:10.774424    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4334bf-5678f547b-cvfkg" event={"ID":"abb9d44c-4965-46de-ad45-3d15b8a09630","Type":"ContainerDied","Data":"af2a594df74707a0a9f10738b672472e4d653b6576c39ba76c382607a3dd1a5f"}
Oct 30 22:55:10.774512 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:55:10.774449    5678 scope.go:122] "RemoveContainer" containerID="dfa7cb30e239e72eb0c901a73d353107b3373821ff4d7e3c5eaa75358f794af4"
Oct 30 22:55:10.786267 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:55:10.786089    5678 scope.go:122] "RemoveContainer" containerID="dfa7cb30e239e72eb0c901a73d353107b3373821ff4d7e3c5eaa75358f794af4"
Oct 30 22:55:10.786608 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:55:10.786581    5678 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfa7cb30e239e72eb0c901a73d353107b3373821ff4d7e3c5eaa75358f794af4\": container with ID starting with dfa7cb30e239e72eb0c901a73d353107b3373821ff4d7e3c5eaa75358f794af4 not found: ID does not exist" containerID="dfa7cb30e239e72eb0c901a73d353107b3373821ff4d7e3c5eaa75358f794af4"
Oct 30 22:55:10.786732 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:55:10.786613    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfa7cb30e239e72eb0c901a73d353107b3373821ff4d7e3c5eaa75358f794af4"} err="failed to get container status \"dfa7cb30e239e72eb0c901a73d353107b3373821ff4d7e3c5eaa75358f794af4\": rpc error: code = NotFound desc = could not find container \"dfa7cb30e239e72eb0c901a73d353107b3373821ff4d7e3c5eaa75358f794af4\": container with ID starting with dfa7cb30e239e72eb0c901a73d353107b3373821ff4d7e3c5eaa75358f794af4 not found: ID does not exist"
Oct 30 22:55:10.930311 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:55:10.930271    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4334bf-5678f547b-cvfkg"]
Oct 30 22:55:10.936519 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:55:10.936496    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4334bf-5678f547b-cvfkg"]
Oct 30 22:55:12.521715 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:55:12.521681    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abb9d44c-4965-46de-ad45-3d15b8a09630" path="/var/lib/kubelet/pods/abb9d44c-4965-46de-ad45-3d15b8a09630/volumes"
Oct 30 22:55:40.045246 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:55:40.045211    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ksd4n"]
Oct 30 22:55:40.092446 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:55:40.092411    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ksd4n"]
Oct 30 22:55:40.521733 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:55:40.521699    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81681875-a3fb-4fc7-b253-940c2370c6cb" path="/var/lib/kubelet/pods/81681875-a3fb-4fc7-b253-940c2370c6cb/volumes"
Oct 30 22:55:40.570727 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:55:40.570693    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 22:56:10.571048 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:56:10.571012    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 22:56:24.687252 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:56:24.687214    5678 scope.go:122] "RemoveContainer" containerID="21699f6bce6d2850c8bac5207c485780307ff4a292e2a065e151fd07abd2f98c"
Oct 30 22:56:24.695788 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:56:24.695763    5678 scope.go:122] "RemoveContainer" containerID="c9c115e52235ba0d30e7ae799869f905bc23ac4219f90abe852cb8cc8bae18be"
Oct 30 22:56:40.571487 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:56:40.571395    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 22:56:43.640243 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:56:43.640203    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-mtqlj"]
Oct 30 22:56:43.640710 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:56:43.640691    5678 cpu_manager.go:414] "containerMap: removing container" podUID="abb9d44c-4965-46de-ad45-3d15b8a09630" containerName="maas-api"
Oct 30 22:56:43.640769 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:56:43.640712    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abb9d44c-4965-46de-ad45-3d15b8a09630" containerName="maas-api"
Oct 30 22:56:43.640817 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:56:43.640782    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="abb9d44c-4965-46de-ad45-3d15b8a09630" containerName="maas-api"
Oct 30 22:56:43.643689 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:56:43.643661    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-mtqlj"
Oct 30 22:56:43.693040 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:56:43.693014    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5rvcj\""
Oct 30 22:56:43.740261 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:56:43.740230    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:56:43.740261 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:56:43.740239    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:56:43.740470 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:56:43.740293    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:56:43.742263 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:56:43.742242    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-mtqlj"]
Oct 30 22:56:43.756670 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:56:43.756647    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4g4ms\" (UniqueName: \"kubernetes.io/projected/5006f08e-1ebc-4e9f-bad6-bac41a147461-kube-api-access-4g4ms\") pod \"maas-api-revoke-sub-452b6d7b-mtqlj\" (UID: \"5006f08e-1ebc-4e9f-bad6-bac41a147461\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-mtqlj"
Oct 30 22:56:43.756779 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:56:43.756711    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5006f08e-1ebc-4e9f-bad6-bac41a147461-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-mtqlj\" (UID: \"5006f08e-1ebc-4e9f-bad6-bac41a147461\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-mtqlj"
Oct 30 22:56:43.756821 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:56:43.756788    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5006f08e-1ebc-4e9f-bad6-bac41a147461-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-mtqlj\" (UID: \"5006f08e-1ebc-4e9f-bad6-bac41a147461\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-mtqlj"
Oct 30 22:56:43.857988 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:56:43.857958    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4g4ms\" (UniqueName: \"kubernetes.io/projected/5006f08e-1ebc-4e9f-bad6-bac41a147461-kube-api-access-4g4ms\") pod \"maas-api-revoke-sub-452b6d7b-mtqlj\" (UID: \"5006f08e-1ebc-4e9f-bad6-bac41a147461\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-mtqlj"
Oct 30 22:56:43.858170 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:56:43.858031    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5006f08e-1ebc-4e9f-bad6-bac41a147461-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-mtqlj\" (UID: \"5006f08e-1ebc-4e9f-bad6-bac41a147461\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-mtqlj"
Oct 30 22:56:43.858170 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:56:43.858054    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5006f08e-1ebc-4e9f-bad6-bac41a147461-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-mtqlj\" (UID: \"5006f08e-1ebc-4e9f-bad6-bac41a147461\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-mtqlj"
Oct 30 22:56:43.858631 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:56:43.858610    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5006f08e-1ebc-4e9f-bad6-bac41a147461-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-mtqlj\" (UID: \"5006f08e-1ebc-4e9f-bad6-bac41a147461\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-mtqlj"
Oct 30 22:56:43.860199 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:56:43.860178    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5006f08e-1ebc-4e9f-bad6-bac41a147461-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-mtqlj\" (UID: \"5006f08e-1ebc-4e9f-bad6-bac41a147461\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-mtqlj"
Oct 30 22:56:43.915907 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:56:43.915853    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4g4ms\" (UniqueName: \"kubernetes.io/projected/5006f08e-1ebc-4e9f-bad6-bac41a147461-kube-api-access-4g4ms\") pod \"maas-api-revoke-sub-452b6d7b-mtqlj\" (UID: \"5006f08e-1ebc-4e9f-bad6-bac41a147461\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-mtqlj"
Oct 30 22:56:43.956933 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:56:43.956907    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-mtqlj"
Oct 30 22:56:44.330962 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:56:44.330939    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-mtqlj"]
Oct 30 22:56:44.333199 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:56:44.333171    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5006f08e_1ebc_4e9f_bad6_bac41a147461.slice/crio-6fa7961918898f8521e8b9fd9277264612f67065a5c8c6785d203b974dc8913b WatchSource:0}: Error finding container 6fa7961918898f8521e8b9fd9277264612f67065a5c8c6785d203b974dc8913b: Status 404 returned error can't find the container with id 6fa7961918898f8521e8b9fd9277264612f67065a5c8c6785d203b974dc8913b
Oct 30 22:56:45.260706 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:56:45.260669    5678 generic.go:356] "Generic (PLEG): container finished" podID="5006f08e-1ebc-4e9f-bad6-bac41a147461" containerID="d00ffd404f3792798fbe72cab767c75d63922ee0d6ce551175bd3074781f8b70" exitCode=0
Oct 30 22:56:45.261141 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:56:45.260721    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-mtqlj" event={"ID":"5006f08e-1ebc-4e9f-bad6-bac41a147461","Type":"ContainerDied","Data":"d00ffd404f3792798fbe72cab767c75d63922ee0d6ce551175bd3074781f8b70"}
Oct 30 22:56:45.261141 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:56:45.260785    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-mtqlj" event={"ID":"5006f08e-1ebc-4e9f-bad6-bac41a147461","Type":"ContainerStarted","Data":"6fa7961918898f8521e8b9fd9277264612f67065a5c8c6785d203b974dc8913b"}
Oct 30 22:56:46.428717 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:56:46.428687    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-mtqlj"
Oct 30 22:56:46.583257 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:56:46.583165    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5006f08e-1ebc-4e9f-bad6-bac41a147461-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5006f08e-1ebc-4e9f-bad6-bac41a147461-maas-api-xks-ca\") pod \"5006f08e-1ebc-4e9f-bad6-bac41a147461\" (UID: \"5006f08e-1ebc-4e9f-bad6-bac41a147461\") "
Oct 30 22:56:46.583461 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:56:46.583333    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5006f08e-1ebc-4e9f-bad6-bac41a147461-kube-api-access-4g4ms\" (UniqueName: \"kubernetes.io/projected/5006f08e-1ebc-4e9f-bad6-bac41a147461-kube-api-access-4g4ms\") pod \"5006f08e-1ebc-4e9f-bad6-bac41a147461\" (UID: \"5006f08e-1ebc-4e9f-bad6-bac41a147461\") "
Oct 30 22:56:46.583461 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:56:46.583413    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5006f08e-1ebc-4e9f-bad6-bac41a147461-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5006f08e-1ebc-4e9f-bad6-bac41a147461-maas-api-openshift-ca\") pod \"5006f08e-1ebc-4e9f-bad6-bac41a147461\" (UID: \"5006f08e-1ebc-4e9f-bad6-bac41a147461\") "
Oct 30 22:56:46.584059 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:56:46.584028    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5006f08e-1ebc-4e9f-bad6-bac41a147461-maas-api-openshift-ca" pod "5006f08e-1ebc-4e9f-bad6-bac41a147461" (UID: "5006f08e-1ebc-4e9f-bad6-bac41a147461"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:56:46.585773 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:56:46.585737    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5006f08e-1ebc-4e9f-bad6-bac41a147461-maas-api-xks-ca" pod "5006f08e-1ebc-4e9f-bad6-bac41a147461" (UID: "5006f08e-1ebc-4e9f-bad6-bac41a147461"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:56:46.585773 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:56:46.585744    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5006f08e-1ebc-4e9f-bad6-bac41a147461-kube-api-access-4g4ms" pod "5006f08e-1ebc-4e9f-bad6-bac41a147461" (UID: "5006f08e-1ebc-4e9f-bad6-bac41a147461"). InnerVolumeSpecName "kube-api-access-4g4ms". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:56:46.684766 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:56:46.684729    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5006f08e-1ebc-4e9f-bad6-bac41a147461-maas-api-xks-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:56:46.685028 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:56:46.685001    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4g4ms\" (UniqueName: \"kubernetes.io/projected/5006f08e-1ebc-4e9f-bad6-bac41a147461-kube-api-access-4g4ms\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:56:46.685028 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:56:46.685036    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5006f08e-1ebc-4e9f-bad6-bac41a147461-maas-api-openshift-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:56:47.293865 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:56:47.293825    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-mtqlj" event={"ID":"5006f08e-1ebc-4e9f-bad6-bac41a147461","Type":"ContainerDied","Data":"6fa7961918898f8521e8b9fd9277264612f67065a5c8c6785d203b974dc8913b"}
Oct 30 22:56:47.294157 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:56:47.294125    5678 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6fa7961918898f8521e8b9fd9277264612f67065a5c8c6785d203b974dc8913b"
Oct 30 22:56:47.294157 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:56:47.293843    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-mtqlj"
Oct 30 22:57:10.569929 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:57:10.569892    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 22:57:12.909491 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:57:12.909462    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-bmtdp"]
Oct 30 22:57:12.909905 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:57:12.909822    5678 cpu_manager.go:414] "containerMap: removing container" podUID="5006f08e-1ebc-4e9f-bad6-bac41a147461" containerName="revoke-keys"
Oct 30 22:57:12.909905 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:57:12.909832    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5006f08e-1ebc-4e9f-bad6-bac41a147461" containerName="revoke-keys"
Oct 30 22:57:12.909905 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:57:12.909901    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="5006f08e-1ebc-4e9f-bad6-bac41a147461" containerName="revoke-keys"
Oct 30 22:57:12.912847 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:57:12.912763    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-bmtdp"
Oct 30 22:57:12.962728 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:57:12.962686    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:57:13.009730 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:57:13.009690    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5rvcj\""
Oct 30 22:57:13.009892 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:57:13.009743    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:57:13.009892 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:57:13.009764    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:57:13.012390 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:57:13.012369    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-bmtdp"]
Oct 30 22:57:13.076940 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:57:13.076906    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56a5a108-7114-47cd-a79c-aa2723254d6d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-bmtdp\" (UID: \"56a5a108-7114-47cd-a79c-aa2723254d6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-bmtdp"
Oct 30 22:57:13.077081 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:57:13.077055    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56a5a108-7114-47cd-a79c-aa2723254d6d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-bmtdp\" (UID: \"56a5a108-7114-47cd-a79c-aa2723254d6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-bmtdp"
Oct 30 22:57:13.077129 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:57:13.077081    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbplp\" (UniqueName: \"kubernetes.io/projected/56a5a108-7114-47cd-a79c-aa2723254d6d-kube-api-access-xbplp\") pod \"maas-api-revoke-sub-3e31a506-bmtdp\" (UID: \"56a5a108-7114-47cd-a79c-aa2723254d6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-bmtdp"
Oct 30 22:57:13.177767 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:57:13.177689    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56a5a108-7114-47cd-a79c-aa2723254d6d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-bmtdp\" (UID: \"56a5a108-7114-47cd-a79c-aa2723254d6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-bmtdp"
Oct 30 22:57:13.177767 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:57:13.177723    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbplp\" (UniqueName: \"kubernetes.io/projected/56a5a108-7114-47cd-a79c-aa2723254d6d-kube-api-access-xbplp\") pod \"maas-api-revoke-sub-3e31a506-bmtdp\" (UID: \"56a5a108-7114-47cd-a79c-aa2723254d6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-bmtdp"
Oct 30 22:57:13.177767 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:57:13.177756    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56a5a108-7114-47cd-a79c-aa2723254d6d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-bmtdp\" (UID: \"56a5a108-7114-47cd-a79c-aa2723254d6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-bmtdp"
Oct 30 22:57:13.178336 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:57:13.178314    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56a5a108-7114-47cd-a79c-aa2723254d6d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-bmtdp\" (UID: \"56a5a108-7114-47cd-a79c-aa2723254d6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-bmtdp"
Oct 30 22:57:13.179989 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:57:13.179963    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56a5a108-7114-47cd-a79c-aa2723254d6d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-bmtdp\" (UID: \"56a5a108-7114-47cd-a79c-aa2723254d6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-bmtdp"
Oct 30 22:57:13.236816 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:57:13.236788    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbplp\" (UniqueName: \"kubernetes.io/projected/56a5a108-7114-47cd-a79c-aa2723254d6d-kube-api-access-xbplp\") pod \"maas-api-revoke-sub-3e31a506-bmtdp\" (UID: \"56a5a108-7114-47cd-a79c-aa2723254d6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-bmtdp"
Oct 30 22:57:13.528911 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:57:13.528821    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-bmtdp"
Oct 30 22:57:13.903101 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:57:13.903075    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-bmtdp"]
Oct 30 22:57:13.904910 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:57:13.904881    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56a5a108_7114_47cd_a79c_aa2723254d6d.slice/crio-3255c2e49b051710e8205de8d078f806f3fed274ffc175576bb8196deb068f76 WatchSource:0}: Error finding container 3255c2e49b051710e8205de8d078f806f3fed274ffc175576bb8196deb068f76: Status 404 returned error can't find the container with id 3255c2e49b051710e8205de8d078f806f3fed274ffc175576bb8196deb068f76
Oct 30 22:57:14.462151 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:57:14.462106    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-bmtdp" event={"ID":"56a5a108-7114-47cd-a79c-aa2723254d6d","Type":"ContainerStarted","Data":"b734b06787af7679f5b44003d86bcec8d39aed068ce1365787283fc06d4b3420"}
Oct 30 22:57:14.462702 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:57:14.462168    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-bmtdp" event={"ID":"56a5a108-7114-47cd-a79c-aa2723254d6d","Type":"ContainerStarted","Data":"3255c2e49b051710e8205de8d078f806f3fed274ffc175576bb8196deb068f76"}
Oct 30 22:57:14.575777 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:57:14.575712    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-bmtdp" podStartSLOduration=2.575693339 podStartE2EDuration="2.575693339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:57:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:57:14.573824011 +0000 UTC m=+4012.178131564" watchObservedRunningTime="2026-10-30 22:57:14.575693339 +0000 UTC m=+4012.180000908"
Oct 30 22:57:15.494200 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:57:15.494161    5678 generic.go:356] "Generic (PLEG): container finished" podID="56a5a108-7114-47cd-a79c-aa2723254d6d" containerID="b734b06787af7679f5b44003d86bcec8d39aed068ce1365787283fc06d4b3420" exitCode=0
Oct 30 22:57:15.494756 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:57:15.494732    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-bmtdp" event={"ID":"56a5a108-7114-47cd-a79c-aa2723254d6d","Type":"ContainerDied","Data":"b734b06787af7679f5b44003d86bcec8d39aed068ce1365787283fc06d4b3420"}
Oct 30 22:57:16.675255 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:57:16.675231    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-bmtdp"
Oct 30 22:57:16.715514 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:57:16.715486    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56a5a108-7114-47cd-a79c-aa2723254d6d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56a5a108-7114-47cd-a79c-aa2723254d6d-maas-api-openshift-ca\") pod \"56a5a108-7114-47cd-a79c-aa2723254d6d\" (UID: \"56a5a108-7114-47cd-a79c-aa2723254d6d\") "
Oct 30 22:57:16.715636 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:57:16.715538    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56a5a108-7114-47cd-a79c-aa2723254d6d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56a5a108-7114-47cd-a79c-aa2723254d6d-maas-api-xks-ca\") pod \"56a5a108-7114-47cd-a79c-aa2723254d6d\" (UID: \"56a5a108-7114-47cd-a79c-aa2723254d6d\") "
Oct 30 22:57:16.715636 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:57:16.715562    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56a5a108-7114-47cd-a79c-aa2723254d6d-kube-api-access-xbplp\" (UniqueName: \"kubernetes.io/projected/56a5a108-7114-47cd-a79c-aa2723254d6d-kube-api-access-xbplp\") pod \"56a5a108-7114-47cd-a79c-aa2723254d6d\" (UID: \"56a5a108-7114-47cd-a79c-aa2723254d6d\") "
Oct 30 22:57:16.715794 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:57:16.715772    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56a5a108-7114-47cd-a79c-aa2723254d6d-maas-api-openshift-ca" pod "56a5a108-7114-47cd-a79c-aa2723254d6d" (UID: "56a5a108-7114-47cd-a79c-aa2723254d6d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:57:16.717613 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:57:16.717582    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56a5a108-7114-47cd-a79c-aa2723254d6d-maas-api-xks-ca" pod "56a5a108-7114-47cd-a79c-aa2723254d6d" (UID: "56a5a108-7114-47cd-a79c-aa2723254d6d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:57:16.717740 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:57:16.717691    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56a5a108-7114-47cd-a79c-aa2723254d6d-kube-api-access-xbplp" pod "56a5a108-7114-47cd-a79c-aa2723254d6d" (UID: "56a5a108-7114-47cd-a79c-aa2723254d6d"). InnerVolumeSpecName "kube-api-access-xbplp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:57:16.816726 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:57:16.816647    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56a5a108-7114-47cd-a79c-aa2723254d6d-maas-api-openshift-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:57:16.816726 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:57:16.816677    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56a5a108-7114-47cd-a79c-aa2723254d6d-maas-api-xks-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:57:16.816726 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:57:16.816691    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbplp\" (UniqueName: \"kubernetes.io/projected/56a5a108-7114-47cd-a79c-aa2723254d6d-kube-api-access-xbplp\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:57:17.507267 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:57:17.507237    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-bmtdp"
Oct 30 22:57:17.507267 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:57:17.507250    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-bmtdp" event={"ID":"56a5a108-7114-47cd-a79c-aa2723254d6d","Type":"ContainerDied","Data":"3255c2e49b051710e8205de8d078f806f3fed274ffc175576bb8196deb068f76"}
Oct 30 22:57:17.507536 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:57:17.507308    5678 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3255c2e49b051710e8205de8d078f806f3fed274ffc175576bb8196deb068f76"
Oct 30 22:57:24.718922 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:57:24.718893    5678 scope.go:122] "RemoveContainer" containerID="29537a03414716df2ec9d6e9a1e03871cf5d1f40e5a02c3dff8e8c32f441b16b"
Oct 30 22:57:40.570228 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:57:40.570192    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 22:57:45.472548 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:57:45.472516    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5b97c83-njjk9"]
Oct 30 22:57:45.472993 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:57:45.472904    5678 cpu_manager.go:414] "containerMap: removing container" podUID="56a5a108-7114-47cd-a79c-aa2723254d6d" containerName="revoke-keys"
Oct 30 22:57:45.472993 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:57:45.472915    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56a5a108-7114-47cd-a79c-aa2723254d6d" containerName="revoke-keys"
Oct 30 22:57:45.472993 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:57:45.472988    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="56a5a108-7114-47cd-a79c-aa2723254d6d" containerName="revoke-keys"
Oct 30 22:57:45.475973 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:57:45.475948    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5b97c83-njjk9"
Oct 30 22:57:45.525939 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:57:45.525911    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:57:45.572708 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:57:45.572677    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:57:45.572871 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:57:45.572712    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:57:45.572871 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:57:45.572858    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5rvcj\""
Oct 30 22:57:45.574727 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:57:45.574709    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5b97c83-njjk9"]
Oct 30 22:57:45.603140 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:57:45.603114    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50d35d93-4c69-4451-a0a8-82a9ee9740bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5b97c83-njjk9\" (UID: \"50d35d93-4c69-4451-a0a8-82a9ee9740bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5b97c83-njjk9"
Oct 30 22:57:45.603293 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:57:45.603185    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50d35d93-4c69-4451-a0a8-82a9ee9740bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5b97c83-njjk9\" (UID: \"50d35d93-4c69-4451-a0a8-82a9ee9740bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5b97c83-njjk9"
Oct 30 22:57:45.603427 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:57:45.603377    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p22jt\" (UniqueName: \"kubernetes.io/projected/50d35d93-4c69-4451-a0a8-82a9ee9740bd-kube-api-access-p22jt\") pod \"maas-api-revoke-sub-b5b97c83-njjk9\" (UID: \"50d35d93-4c69-4451-a0a8-82a9ee9740bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5b97c83-njjk9"
Oct 30 22:57:45.704946 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:57:45.704912    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50d35d93-4c69-4451-a0a8-82a9ee9740bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5b97c83-njjk9\" (UID: \"50d35d93-4c69-4451-a0a8-82a9ee9740bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5b97c83-njjk9"
Oct 30 22:57:45.705123 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:57:45.704985    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50d35d93-4c69-4451-a0a8-82a9ee9740bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5b97c83-njjk9\" (UID: \"50d35d93-4c69-4451-a0a8-82a9ee9740bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5b97c83-njjk9"
Oct 30 22:57:45.705123 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:57:45.705016    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p22jt\" (UniqueName: \"kubernetes.io/projected/50d35d93-4c69-4451-a0a8-82a9ee9740bd-kube-api-access-p22jt\") pod \"maas-api-revoke-sub-b5b97c83-njjk9\" (UID: \"50d35d93-4c69-4451-a0a8-82a9ee9740bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5b97c83-njjk9"
Oct 30 22:57:45.705744 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:57:45.705723    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50d35d93-4c69-4451-a0a8-82a9ee9740bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5b97c83-njjk9\" (UID: \"50d35d93-4c69-4451-a0a8-82a9ee9740bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5b97c83-njjk9"
Oct 30 22:57:45.707260 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:57:45.707241    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50d35d93-4c69-4451-a0a8-82a9ee9740bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5b97c83-njjk9\" (UID: \"50d35d93-4c69-4451-a0a8-82a9ee9740bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5b97c83-njjk9"
Oct 30 22:57:45.763893 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:57:45.763827    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p22jt\" (UniqueName: \"kubernetes.io/projected/50d35d93-4c69-4451-a0a8-82a9ee9740bd-kube-api-access-p22jt\") pod \"maas-api-revoke-sub-b5b97c83-njjk9\" (UID: \"50d35d93-4c69-4451-a0a8-82a9ee9740bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5b97c83-njjk9"
Oct 30 22:57:45.788789 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:57:45.788762    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5b97c83-njjk9"
Oct 30 22:57:46.166822 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:57:46.166793    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5b97c83-njjk9"]
Oct 30 22:57:46.167904 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:57:46.167877    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50d35d93_4c69_4451_a0a8_82a9ee9740bd.slice/crio-04a53bf4eda9b877a92fa0e96270b5e00dc8120aa54aee87f03a1e6883971b61 WatchSource:0}: Error finding container 04a53bf4eda9b877a92fa0e96270b5e00dc8120aa54aee87f03a1e6883971b61: Status 404 returned error can't find the container with id 04a53bf4eda9b877a92fa0e96270b5e00dc8120aa54aee87f03a1e6883971b61
Oct 30 22:57:46.736887 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:57:46.736854    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5b97c83-njjk9" event={"ID":"50d35d93-4c69-4451-a0a8-82a9ee9740bd","Type":"ContainerStarted","Data":"c548848b7ffe53cd6cdb2abdaeb759552b1b79d28b412b51ce8d40226685ccb9"}
Oct 30 22:57:46.737362 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:57:46.736902    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5b97c83-njjk9" event={"ID":"50d35d93-4c69-4451-a0a8-82a9ee9740bd","Type":"ContainerStarted","Data":"04a53bf4eda9b877a92fa0e96270b5e00dc8120aa54aee87f03a1e6883971b61"}
Oct 30 22:57:47.743032 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:57:47.742996    5678 generic.go:356] "Generic (PLEG): container finished" podID="50d35d93-4c69-4451-a0a8-82a9ee9740bd" containerID="c548848b7ffe53cd6cdb2abdaeb759552b1b79d28b412b51ce8d40226685ccb9" exitCode=0
Oct 30 22:57:47.743469 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:57:47.743039    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5b97c83-njjk9" event={"ID":"50d35d93-4c69-4451-a0a8-82a9ee9740bd","Type":"ContainerDied","Data":"c548848b7ffe53cd6cdb2abdaeb759552b1b79d28b412b51ce8d40226685ccb9"}
Oct 30 22:57:48.878678 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:57:48.878653    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5b97c83-njjk9"
Oct 30 22:57:48.936782 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:57:48.936757    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50d35d93-4c69-4451-a0a8-82a9ee9740bd-kube-api-access-p22jt\" (UniqueName: \"kubernetes.io/projected/50d35d93-4c69-4451-a0a8-82a9ee9740bd-kube-api-access-p22jt\") pod \"50d35d93-4c69-4451-a0a8-82a9ee9740bd\" (UID: \"50d35d93-4c69-4451-a0a8-82a9ee9740bd\") "
Oct 30 22:57:48.936916 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:57:48.936796    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50d35d93-4c69-4451-a0a8-82a9ee9740bd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50d35d93-4c69-4451-a0a8-82a9ee9740bd-maas-api-xks-ca\") pod \"50d35d93-4c69-4451-a0a8-82a9ee9740bd\" (UID: \"50d35d93-4c69-4451-a0a8-82a9ee9740bd\") "
Oct 30 22:57:48.936916 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:57:48.936841    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50d35d93-4c69-4451-a0a8-82a9ee9740bd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50d35d93-4c69-4451-a0a8-82a9ee9740bd-maas-api-openshift-ca\") pod \"50d35d93-4c69-4451-a0a8-82a9ee9740bd\" (UID: \"50d35d93-4c69-4451-a0a8-82a9ee9740bd\") "
Oct 30 22:57:48.937220 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:57:48.937199    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50d35d93-4c69-4451-a0a8-82a9ee9740bd-maas-api-openshift-ca" pod "50d35d93-4c69-4451-a0a8-82a9ee9740bd" (UID: "50d35d93-4c69-4451-a0a8-82a9ee9740bd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:57:48.938829 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:57:48.938806    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50d35d93-4c69-4451-a0a8-82a9ee9740bd-maas-api-xks-ca" pod "50d35d93-4c69-4451-a0a8-82a9ee9740bd" (UID: "50d35d93-4c69-4451-a0a8-82a9ee9740bd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:57:48.938949 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:57:48.938859    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50d35d93-4c69-4451-a0a8-82a9ee9740bd-kube-api-access-p22jt" pod "50d35d93-4c69-4451-a0a8-82a9ee9740bd" (UID: "50d35d93-4c69-4451-a0a8-82a9ee9740bd"). InnerVolumeSpecName "kube-api-access-p22jt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:57:49.038125 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:57:49.038062    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p22jt\" (UniqueName: \"kubernetes.io/projected/50d35d93-4c69-4451-a0a8-82a9ee9740bd-kube-api-access-p22jt\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:57:49.038125 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:57:49.038086    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50d35d93-4c69-4451-a0a8-82a9ee9740bd-maas-api-xks-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:57:49.038125 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:57:49.038096    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50d35d93-4c69-4451-a0a8-82a9ee9740bd-maas-api-openshift-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:57:49.755257 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:57:49.755225    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5b97c83-njjk9" event={"ID":"50d35d93-4c69-4451-a0a8-82a9ee9740bd","Type":"ContainerDied","Data":"04a53bf4eda9b877a92fa0e96270b5e00dc8120aa54aee87f03a1e6883971b61"}
Oct 30 22:57:49.755453 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:57:49.755278    5678 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="04a53bf4eda9b877a92fa0e96270b5e00dc8120aa54aee87f03a1e6883971b61"
Oct 30 22:57:49.755453 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:57:49.755241    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5b97c83-njjk9"
Oct 30 22:57:50.051182 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:57:50.051095    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-h9jv7"]
Oct 30 22:57:50.080653 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:57:50.080626    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-h9jv7"]
Oct 30 22:57:50.521458 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:57:50.521423    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="331b8c93-8ca7-4cda-8364-e146dda3aa33" path="/var/lib/kubelet/pods/331b8c93-8ca7-4cda-8364-e146dda3aa33/volumes"
Oct 30 22:57:58.919871 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:57:58.919833    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7f4ccfcf79-94lpf"]
Oct 30 22:57:58.920389 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:57:58.920370    5678 cpu_manager.go:414] "containerMap: removing container" podUID="50d35d93-4c69-4451-a0a8-82a9ee9740bd" containerName="revoke-keys"
Oct 30 22:57:58.920529 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:57:58.920391    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50d35d93-4c69-4451-a0a8-82a9ee9740bd" containerName="revoke-keys"
Oct 30 22:57:58.920529 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:57:58.920523    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="50d35d93-4c69-4451-a0a8-82a9ee9740bd" containerName="revoke-keys"
Oct 30 22:57:58.923690 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:57:58.923668    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7f4ccfcf79-94lpf"
Oct 30 22:57:59.021382 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:57:59.021349    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:57:59.021636 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:57:59.021348    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:57:59.021636 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:57:59.021363    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 30 22:57:59.021843 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:57:59.021638    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-jglqk\""
Oct 30 22:57:59.021843 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:57:59.021491    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Oct 30 22:57:59.024023 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:57:59.024002    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7f4ccfcf79-94lpf"]
Oct 30 22:57:59.053217 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:57:59.053176    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83715f01-2c04-48e4-af71-594ec7a93320-maas-api-tls\") pod \"maas-api-7f4ccfcf79-94lpf\" (UID: \"83715f01-2c04-48e4-af71-594ec7a93320\") " pod="odh-ai-gateway-infra/maas-api-7f4ccfcf79-94lpf"
Oct 30 22:57:59.053428 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:57:59.053225    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83715f01-2c04-48e4-af71-594ec7a93320-metrics-certs\") pod \"maas-api-7f4ccfcf79-94lpf\" (UID: \"83715f01-2c04-48e4-af71-594ec7a93320\") " pod="odh-ai-gateway-infra/maas-api-7f4ccfcf79-94lpf"
Oct 30 22:57:59.053428 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:57:59.053265    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83715f01-2c04-48e4-af71-594ec7a93320-openshift-service-ca\") pod \"maas-api-7f4ccfcf79-94lpf\" (UID: \"83715f01-2c04-48e4-af71-594ec7a93320\") " pod="odh-ai-gateway-infra/maas-api-7f4ccfcf79-94lpf"
Oct 30 22:57:59.053428 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:57:59.053360    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dk4k5\" (UniqueName: \"kubernetes.io/projected/83715f01-2c04-48e4-af71-594ec7a93320-kube-api-access-dk4k5\") pod \"maas-api-7f4ccfcf79-94lpf\" (UID: \"83715f01-2c04-48e4-af71-594ec7a93320\") " pod="odh-ai-gateway-infra/maas-api-7f4ccfcf79-94lpf"
Oct 30 22:57:59.154855 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:57:59.154820    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83715f01-2c04-48e4-af71-594ec7a93320-maas-api-tls\") pod \"maas-api-7f4ccfcf79-94lpf\" (UID: \"83715f01-2c04-48e4-af71-594ec7a93320\") " pod="odh-ai-gateway-infra/maas-api-7f4ccfcf79-94lpf"
Oct 30 22:57:59.155019 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:57:59.154862    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83715f01-2c04-48e4-af71-594ec7a93320-metrics-certs\") pod \"maas-api-7f4ccfcf79-94lpf\" (UID: \"83715f01-2c04-48e4-af71-594ec7a93320\") " pod="odh-ai-gateway-infra/maas-api-7f4ccfcf79-94lpf"
Oct 30 22:57:59.155019 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:57:59.154980    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83715f01-2c04-48e4-af71-594ec7a93320-openshift-service-ca\") pod \"maas-api-7f4ccfcf79-94lpf\" (UID: \"83715f01-2c04-48e4-af71-594ec7a93320\") " pod="odh-ai-gateway-infra/maas-api-7f4ccfcf79-94lpf"
Oct 30 22:57:59.155096 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:57:59.155081    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dk4k5\" (UniqueName: \"kubernetes.io/projected/83715f01-2c04-48e4-af71-594ec7a93320-kube-api-access-dk4k5\") pod \"maas-api-7f4ccfcf79-94lpf\" (UID: \"83715f01-2c04-48e4-af71-594ec7a93320\") " pod="odh-ai-gateway-infra/maas-api-7f4ccfcf79-94lpf"
Oct 30 22:57:59.155659 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:57:59.155637    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83715f01-2c04-48e4-af71-594ec7a93320-openshift-service-ca\") pod \"maas-api-7f4ccfcf79-94lpf\" (UID: \"83715f01-2c04-48e4-af71-594ec7a93320\") " pod="odh-ai-gateway-infra/maas-api-7f4ccfcf79-94lpf"
Oct 30 22:57:59.157277 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:57:59.157255    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83715f01-2c04-48e4-af71-594ec7a93320-metrics-certs\") pod \"maas-api-7f4ccfcf79-94lpf\" (UID: \"83715f01-2c04-48e4-af71-594ec7a93320\") " pod="odh-ai-gateway-infra/maas-api-7f4ccfcf79-94lpf"
Oct 30 22:57:59.157464 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:57:59.157443    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83715f01-2c04-48e4-af71-594ec7a93320-maas-api-tls\") pod \"maas-api-7f4ccfcf79-94lpf\" (UID: \"83715f01-2c04-48e4-af71-594ec7a93320\") " pod="odh-ai-gateway-infra/maas-api-7f4ccfcf79-94lpf"
Oct 30 22:57:59.214320 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:57:59.214258    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dk4k5\" (UniqueName: \"kubernetes.io/projected/83715f01-2c04-48e4-af71-594ec7a93320-kube-api-access-dk4k5\") pod \"maas-api-7f4ccfcf79-94lpf\" (UID: \"83715f01-2c04-48e4-af71-594ec7a93320\") " pod="odh-ai-gateway-infra/maas-api-7f4ccfcf79-94lpf"
Oct 30 22:57:59.236146 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:57:59.236123    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7f4ccfcf79-94lpf"
Oct 30 22:57:59.613359 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:57:59.613336    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7f4ccfcf79-94lpf"]
Oct 30 22:57:59.614396 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:57:59.614369    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83715f01_2c04_48e4_af71_594ec7a93320.slice/crio-8b2c7af108615b5bc8a95ff5a509cb823582df1fe9893670f417ba57d06e46c5 WatchSource:0}: Error finding container 8b2c7af108615b5bc8a95ff5a509cb823582df1fe9893670f417ba57d06e46c5: Status 404 returned error can't find the container with id 8b2c7af108615b5bc8a95ff5a509cb823582df1fe9893670f417ba57d06e46c5
Oct 30 22:57:59.805958 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:57:59.805925    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7f4ccfcf79-94lpf" event={"ID":"83715f01-2c04-48e4-af71-594ec7a93320","Type":"ContainerStarted","Data":"8b2c7af108615b5bc8a95ff5a509cb823582df1fe9893670f417ba57d06e46c5"}
Oct 30 22:58:00.813948 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:58:00.813911    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7f4ccfcf79-94lpf" event={"ID":"83715f01-2c04-48e4-af71-594ec7a93320","Type":"ContainerStarted","Data":"96133d04b863a603d3bbc43c8d91b1b3cd1891eecdcdc254685e061c3420b2a0"}
Oct 30 22:58:00.814363 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:58:00.814020    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7f4ccfcf79-94lpf"
Oct 30 22:58:00.927822 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:58:00.927767    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7f4ccfcf79-94lpf" podStartSLOduration=1.929581798 podStartE2EDuration="2.927754974s" totalImagesPullingTime="998.173176ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:57:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:58:00.92688359 +0000 UTC m=+4058.531191144" watchObservedRunningTime="2026-10-30 22:58:00.927754974 +0000 UTC m=+4058.532062527"
Oct 30 22:58:06.823753 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:58:06.823720    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7f4ccfcf79-94lpf"
Oct 30 22:58:07.622494 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:58:07.622461    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-57df996b79-zkxjh"]
Oct 30 22:58:07.626177 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:58:07.626153    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-57df996b79-zkxjh"
Oct 30 22:58:07.630498 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:58:07.630475    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/957947a4-ac1a-4dcd-aafa-e79848117b51-metrics-certs\") pod \"maas-api-57df996b79-zkxjh\" (UID: \"957947a4-ac1a-4dcd-aafa-e79848117b51\") " pod="odh-ai-gateway-infra/maas-api-57df996b79-zkxjh"
Oct 30 22:58:07.630655 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:58:07.630574    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/957947a4-ac1a-4dcd-aafa-e79848117b51-openshift-service-ca\") pod \"maas-api-57df996b79-zkxjh\" (UID: \"957947a4-ac1a-4dcd-aafa-e79848117b51\") " pod="odh-ai-gateway-infra/maas-api-57df996b79-zkxjh"
Oct 30 22:58:07.630655 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:58:07.630605    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/957947a4-ac1a-4dcd-aafa-e79848117b51-maas-api-tls\") pod \"maas-api-57df996b79-zkxjh\" (UID: \"957947a4-ac1a-4dcd-aafa-e79848117b51\") " pod="odh-ai-gateway-infra/maas-api-57df996b79-zkxjh"
Oct 30 22:58:07.630778 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:58:07.630663    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nlqh\" (UniqueName: \"kubernetes.io/projected/957947a4-ac1a-4dcd-aafa-e79848117b51-kube-api-access-8nlqh\") pod \"maas-api-57df996b79-zkxjh\" (UID: \"957947a4-ac1a-4dcd-aafa-e79848117b51\") " pod="odh-ai-gateway-infra/maas-api-57df996b79-zkxjh"
Oct 30 22:58:07.725441 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:58:07.725410    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-57df996b79-zkxjh"]
Oct 30 22:58:07.731715 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:58:07.731693    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/957947a4-ac1a-4dcd-aafa-e79848117b51-metrics-certs\") pod \"maas-api-57df996b79-zkxjh\" (UID: \"957947a4-ac1a-4dcd-aafa-e79848117b51\") " pod="odh-ai-gateway-infra/maas-api-57df996b79-zkxjh"
Oct 30 22:58:07.731808 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:58:07.731779    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/957947a4-ac1a-4dcd-aafa-e79848117b51-openshift-service-ca\") pod \"maas-api-57df996b79-zkxjh\" (UID: \"957947a4-ac1a-4dcd-aafa-e79848117b51\") " pod="odh-ai-gateway-infra/maas-api-57df996b79-zkxjh"
Oct 30 22:58:07.731808 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:58:07.731801    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/957947a4-ac1a-4dcd-aafa-e79848117b51-maas-api-tls\") pod \"maas-api-57df996b79-zkxjh\" (UID: \"957947a4-ac1a-4dcd-aafa-e79848117b51\") " pod="odh-ai-gateway-infra/maas-api-57df996b79-zkxjh"
Oct 30 22:58:07.731894 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:58:07.731824    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nlqh\" (UniqueName: \"kubernetes.io/projected/957947a4-ac1a-4dcd-aafa-e79848117b51-kube-api-access-8nlqh\") pod \"maas-api-57df996b79-zkxjh\" (UID: \"957947a4-ac1a-4dcd-aafa-e79848117b51\") " pod="odh-ai-gateway-infra/maas-api-57df996b79-zkxjh"
Oct 30 22:58:07.732507 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:58:07.732484    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/957947a4-ac1a-4dcd-aafa-e79848117b51-openshift-service-ca\") pod \"maas-api-57df996b79-zkxjh\" (UID: \"957947a4-ac1a-4dcd-aafa-e79848117b51\") " pod="odh-ai-gateway-infra/maas-api-57df996b79-zkxjh"
Oct 30 22:58:07.733980 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:58:07.733958    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/957947a4-ac1a-4dcd-aafa-e79848117b51-metrics-certs\") pod \"maas-api-57df996b79-zkxjh\" (UID: \"957947a4-ac1a-4dcd-aafa-e79848117b51\") " pod="odh-ai-gateway-infra/maas-api-57df996b79-zkxjh"
Oct 30 22:58:07.734220 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:58:07.734194    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/957947a4-ac1a-4dcd-aafa-e79848117b51-maas-api-tls\") pod \"maas-api-57df996b79-zkxjh\" (UID: \"957947a4-ac1a-4dcd-aafa-e79848117b51\") " pod="odh-ai-gateway-infra/maas-api-57df996b79-zkxjh"
Oct 30 22:58:07.790700 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:58:07.790668    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nlqh\" (UniqueName: \"kubernetes.io/projected/957947a4-ac1a-4dcd-aafa-e79848117b51-kube-api-access-8nlqh\") pod \"maas-api-57df996b79-zkxjh\" (UID: \"957947a4-ac1a-4dcd-aafa-e79848117b51\") " pod="odh-ai-gateway-infra/maas-api-57df996b79-zkxjh"
Oct 30 22:58:07.940074 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:58:07.939980    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-57df996b79-zkxjh"
Oct 30 22:58:08.317498 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:58:08.317473    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-57df996b79-zkxjh"]
Oct 30 22:58:08.319392 ip-10-0-128-245 kubenswrapper[5678]: W1030 22:58:08.319361    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod957947a4_ac1a_4dcd_aafa_e79848117b51.slice/crio-d885d152a3a981dde748a39493a8299e9bc8ad8ef50b4974ac38f8983b465e88 WatchSource:0}: Error finding container d885d152a3a981dde748a39493a8299e9bc8ad8ef50b4974ac38f8983b465e88: Status 404 returned error can't find the container with id d885d152a3a981dde748a39493a8299e9bc8ad8ef50b4974ac38f8983b465e88
Oct 30 22:58:08.854276 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:58:08.854239    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-57df996b79-zkxjh" event={"ID":"957947a4-ac1a-4dcd-aafa-e79848117b51","Type":"ContainerStarted","Data":"d885d152a3a981dde748a39493a8299e9bc8ad8ef50b4974ac38f8983b465e88"}
Oct 30 22:58:09.860207 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:58:09.860163    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-57df996b79-zkxjh" event={"ID":"957947a4-ac1a-4dcd-aafa-e79848117b51","Type":"ContainerStarted","Data":"d761b01c5861c4c28268d70149c908d186e069c42a955b6fda8a0b207188c0b7"}
Oct 30 22:58:09.860625 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:58:09.860239    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-57df996b79-zkxjh"
Oct 30 22:58:10.571590 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:58:10.571553    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 22:58:15.870695 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:58:15.870667    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-57df996b79-zkxjh"
Oct 30 22:58:15.985757 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:58:15.985704    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-57df996b79-zkxjh" podStartSLOduration=8.001276538 podStartE2EDuration="8.985690942s" totalImagesPullingTime="984.414404ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:58:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:58:09.982294455 +0000 UTC m=+4067.586602009" watchObservedRunningTime="2026-10-30 22:58:15.985690942 +0000 UTC m=+4073.589998494"
Oct 30 22:58:16.033282 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:58:16.033253    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7f4ccfcf79-94lpf"]
Oct 30 22:58:16.033523 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:58:16.033496    5678 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-7f4ccfcf79-94lpf" podUID="83715f01-2c04-48e4-af71-594ec7a93320" containerName="maas-api" containerID="cri-o://96133d04b863a603d3bbc43c8d91b1b3cd1891eecdcdc254685e061c3420b2a0" gracePeriod=30
Oct 30 22:58:16.281809 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:58:16.281779    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7f4ccfcf79-94lpf"
Oct 30 22:58:16.317238 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:58:16.317211    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83715f01-2c04-48e4-af71-594ec7a93320-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83715f01-2c04-48e4-af71-594ec7a93320-openshift-service-ca\") pod \"83715f01-2c04-48e4-af71-594ec7a93320\" (UID: \"83715f01-2c04-48e4-af71-594ec7a93320\") "
Oct 30 22:58:16.317383 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:58:16.317251    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83715f01-2c04-48e4-af71-594ec7a93320-kube-api-access-dk4k5\" (UniqueName: \"kubernetes.io/projected/83715f01-2c04-48e4-af71-594ec7a93320-kube-api-access-dk4k5\") pod \"83715f01-2c04-48e4-af71-594ec7a93320\" (UID: \"83715f01-2c04-48e4-af71-594ec7a93320\") "
Oct 30 22:58:16.317383 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:58:16.317300    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/83715f01-2c04-48e4-af71-594ec7a93320-metrics-certs\" (UniqueName: \"kubernetes.io/secret/83715f01-2c04-48e4-af71-594ec7a93320-metrics-certs\") pod \"83715f01-2c04-48e4-af71-594ec7a93320\" (UID: \"83715f01-2c04-48e4-af71-594ec7a93320\") "
Oct 30 22:58:16.317383 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:58:16.317322    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/83715f01-2c04-48e4-af71-594ec7a93320-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83715f01-2c04-48e4-af71-594ec7a93320-maas-api-tls\") pod \"83715f01-2c04-48e4-af71-594ec7a93320\" (UID: \"83715f01-2c04-48e4-af71-594ec7a93320\") "
Oct 30 22:58:16.317601 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:58:16.317579    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83715f01-2c04-48e4-af71-594ec7a93320-openshift-service-ca" pod "83715f01-2c04-48e4-af71-594ec7a93320" (UID: "83715f01-2c04-48e4-af71-594ec7a93320"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:58:16.317712 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:58:16.317697    5678 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83715f01-2c04-48e4-af71-594ec7a93320-openshift-service-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:58:16.319338 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:58:16.319304    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83715f01-2c04-48e4-af71-594ec7a93320-metrics-certs" pod "83715f01-2c04-48e4-af71-594ec7a93320" (UID: "83715f01-2c04-48e4-af71-594ec7a93320"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:58:16.319338 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:58:16.319333    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83715f01-2c04-48e4-af71-594ec7a93320-maas-api-tls" pod "83715f01-2c04-48e4-af71-594ec7a93320" (UID: "83715f01-2c04-48e4-af71-594ec7a93320"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:58:16.319523 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:58:16.319426    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83715f01-2c04-48e4-af71-594ec7a93320-kube-api-access-dk4k5" pod "83715f01-2c04-48e4-af71-594ec7a93320" (UID: "83715f01-2c04-48e4-af71-594ec7a93320"). InnerVolumeSpecName "kube-api-access-dk4k5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:58:16.418418 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:58:16.418334    5678 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83715f01-2c04-48e4-af71-594ec7a93320-metrics-certs\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:58:16.418418 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:58:16.418359    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83715f01-2c04-48e4-af71-594ec7a93320-maas-api-tls\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:58:16.418418 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:58:16.418371    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dk4k5\" (UniqueName: \"kubernetes.io/projected/83715f01-2c04-48e4-af71-594ec7a93320-kube-api-access-dk4k5\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:58:16.896748 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:58:16.896712    5678 generic.go:356] "Generic (PLEG): container finished" podID="83715f01-2c04-48e4-af71-594ec7a93320" containerID="96133d04b863a603d3bbc43c8d91b1b3cd1891eecdcdc254685e061c3420b2a0" exitCode=0
Oct 30 22:58:16.897225 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:58:16.896759    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7f4ccfcf79-94lpf"
Oct 30 22:58:16.897225 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:58:16.896794    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7f4ccfcf79-94lpf" event={"ID":"83715f01-2c04-48e4-af71-594ec7a93320","Type":"ContainerDied","Data":"96133d04b863a603d3bbc43c8d91b1b3cd1891eecdcdc254685e061c3420b2a0"}
Oct 30 22:58:16.897225 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:58:16.896846    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7f4ccfcf79-94lpf" event={"ID":"83715f01-2c04-48e4-af71-594ec7a93320","Type":"ContainerDied","Data":"8b2c7af108615b5bc8a95ff5a509cb823582df1fe9893670f417ba57d06e46c5"}
Oct 30 22:58:16.897225 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:58:16.896874    5678 scope.go:122] "RemoveContainer" containerID="96133d04b863a603d3bbc43c8d91b1b3cd1891eecdcdc254685e061c3420b2a0"
Oct 30 22:58:16.906128 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:58:16.906103    5678 scope.go:122] "RemoveContainer" containerID="96133d04b863a603d3bbc43c8d91b1b3cd1891eecdcdc254685e061c3420b2a0"
Oct 30 22:58:16.906448 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:58:16.906424    5678 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96133d04b863a603d3bbc43c8d91b1b3cd1891eecdcdc254685e061c3420b2a0\": container with ID starting with 96133d04b863a603d3bbc43c8d91b1b3cd1891eecdcdc254685e061c3420b2a0 not found: ID does not exist" containerID="96133d04b863a603d3bbc43c8d91b1b3cd1891eecdcdc254685e061c3420b2a0"
Oct 30 22:58:16.906528 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:58:16.906462    5678 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96133d04b863a603d3bbc43c8d91b1b3cd1891eecdcdc254685e061c3420b2a0"} err="failed to get container status \"96133d04b863a603d3bbc43c8d91b1b3cd1891eecdcdc254685e061c3420b2a0\": rpc error: code = NotFound desc = could not find container \"96133d04b863a603d3bbc43c8d91b1b3cd1891eecdcdc254685e061c3420b2a0\": container with ID starting with 96133d04b863a603d3bbc43c8d91b1b3cd1891eecdcdc254685e061c3420b2a0 not found: ID does not exist"
Oct 30 22:58:17.062900 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:58:17.062866    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7f4ccfcf79-94lpf"]
Oct 30 22:58:17.066719 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:58:17.066691    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-7f4ccfcf79-94lpf"]
Oct 30 22:58:18.521243 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:58:18.521208    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83715f01-2c04-48e4-af71-594ec7a93320" path="/var/lib/kubelet/pods/83715f01-2c04-48e4-af71-594ec7a93320/volumes"
Oct 30 22:58:19.048703 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:58:19.048672    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-pmz6j"]
Oct 30 22:58:19.095787 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:58:19.095757    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-pmz6j"]
Oct 30 22:58:20.521872 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:58:20.521843    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ded5b225-d97d-4bff-be9a-dd752ddb1edf" path="/var/lib/kubelet/pods/ded5b225-d97d-4bff-be9a-dd752ddb1edf/volumes"
Oct 30 22:58:40.571138 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:58:40.571103    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 22:59:10.569996 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:59:10.569959    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 22:59:24.751761 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:59:24.751730    5678 scope.go:122] "RemoveContainer" containerID="14c84c7fe4662789b34bc6c869214ff6d76b57497b7e243e3964af905578dba1"
Oct 30 22:59:24.760376 ip-10-0-128-245 kubenswrapper[5678]: I1030 22:59:24.760357    5678 scope.go:122] "RemoveContainer" containerID="c4bec6f57816c7e62c8a8093244b6427c0c83a9a228f0030eda1b50a7ff08675"
Oct 30 22:59:40.571133 ip-10-0-128-245 kubenswrapper[5678]: E1030 22:59:40.571043    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 23:00:00.182989 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:00:00.182949    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890020-vktnq"]
Oct 30 23:00:00.183812 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:00:00.183785    5678 cpu_manager.go:414] "containerMap: removing container" podUID="83715f01-2c04-48e4-af71-594ec7a93320" containerName="maas-api"
Oct 30 23:00:00.183812 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:00:00.183809    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83715f01-2c04-48e4-af71-594ec7a93320" containerName="maas-api"
Oct 30 23:00:00.183976 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:00:00.183900    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="83715f01-2c04-48e4-af71-594ec7a93320" containerName="maas-api"
Oct 30 23:00:00.186961 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:00:00.186945    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890020-vktnq"
Oct 30 23:00:00.237932 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:00:00.237897    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k2h55\""
Oct 30 23:00:00.286656 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:00:00.286625    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890020-vktnq"]
Oct 30 23:00:00.297161 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:00:00.297133    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fc370b8-182b-4f2d-9731-8810be2bac99-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890020-vktnq\" (UID: \"5fc370b8-182b-4f2d-9731-8810be2bac99\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890020-vktnq"
Oct 30 23:00:00.297346 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:00:00.297188    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmr6r\" (UniqueName: \"kubernetes.io/projected/5fc370b8-182b-4f2d-9731-8810be2bac99-kube-api-access-nmr6r\") pod \"maas-api-key-cleanup-29890020-vktnq\" (UID: \"5fc370b8-182b-4f2d-9731-8810be2bac99\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890020-vktnq"
Oct 30 23:00:00.397856 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:00:00.397813    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fc370b8-182b-4f2d-9731-8810be2bac99-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890020-vktnq\" (UID: \"5fc370b8-182b-4f2d-9731-8810be2bac99\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890020-vktnq"
Oct 30 23:00:00.397856 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:00:00.397872    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmr6r\" (UniqueName: \"kubernetes.io/projected/5fc370b8-182b-4f2d-9731-8810be2bac99-kube-api-access-nmr6r\") pod \"maas-api-key-cleanup-29890020-vktnq\" (UID: \"5fc370b8-182b-4f2d-9731-8810be2bac99\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890020-vktnq"
Oct 30 23:00:00.398496 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:00:00.398475    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fc370b8-182b-4f2d-9731-8810be2bac99-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890020-vktnq\" (UID: \"5fc370b8-182b-4f2d-9731-8810be2bac99\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890020-vktnq"
Oct 30 23:00:00.457078 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:00:00.457006    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmr6r\" (UniqueName: \"kubernetes.io/projected/5fc370b8-182b-4f2d-9731-8810be2bac99-kube-api-access-nmr6r\") pod \"maas-api-key-cleanup-29890020-vktnq\" (UID: \"5fc370b8-182b-4f2d-9731-8810be2bac99\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890020-vktnq"
Oct 30 23:00:00.500158 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:00:00.500117    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890020-vktnq"
Oct 30 23:00:00.877863 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:00:00.877840    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890020-vktnq"]
Oct 30 23:00:00.880019 ip-10-0-128-245 kubenswrapper[5678]: W1030 23:00:00.879991    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fc370b8_182b_4f2d_9731_8810be2bac99.slice/crio-396acd9b9cbf9eadd38744a828781dc906c7960e5044026bba64f92d326a1bd0 WatchSource:0}: Error finding container 396acd9b9cbf9eadd38744a828781dc906c7960e5044026bba64f92d326a1bd0: Status 404 returned error can't find the container with id 396acd9b9cbf9eadd38744a828781dc906c7960e5044026bba64f92d326a1bd0
Oct 30 23:00:01.461190 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:00:01.461104    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890020-vktnq" event={"ID":"5fc370b8-182b-4f2d-9731-8810be2bac99","Type":"ContainerStarted","Data":"5e1679c29f23e90574d2b098e2e96de5a0845f2724db17f75bfddd453084c6a1"}
Oct 30 23:00:01.461190 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:00:01.461155    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890020-vktnq" event={"ID":"5fc370b8-182b-4f2d-9731-8810be2bac99","Type":"ContainerStarted","Data":"396acd9b9cbf9eadd38744a828781dc906c7960e5044026bba64f92d326a1bd0"}
Oct 30 23:00:01.573111 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:00:01.573056    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890020-vktnq" podStartSLOduration=1.573043557 podStartE2EDuration="1.573043557s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:00:01.571081899 +0000 UTC m=+4179.175389451" watchObservedRunningTime="2026-10-30 23:00:01.573043557 +0000 UTC m=+4179.177351109"
Oct 30 23:00:02.467861 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:00:02.467824    5678 generic.go:356] "Generic (PLEG): container finished" podID="5fc370b8-182b-4f2d-9731-8810be2bac99" containerID="5e1679c29f23e90574d2b098e2e96de5a0845f2724db17f75bfddd453084c6a1" exitCode=60
Oct 30 23:00:02.468255 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:00:02.467886    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890020-vktnq" event={"ID":"5fc370b8-182b-4f2d-9731-8810be2bac99","Type":"ContainerDied","Data":"5e1679c29f23e90574d2b098e2e96de5a0845f2724db17f75bfddd453084c6a1"}
Oct 30 23:00:02.468255 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:00:02.468175    5678 scope.go:122] "RemoveContainer" containerID="5e1679c29f23e90574d2b098e2e96de5a0845f2724db17f75bfddd453084c6a1"
Oct 30 23:00:03.475658 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:00:03.475626    5678 generic.go:356] "Generic (PLEG): container finished" podID="5fc370b8-182b-4f2d-9731-8810be2bac99" containerID="eb4d228d607ed9cacc15fec134b78f595d10b6e6f38b3762b92c02ea711b500c" exitCode=60
Oct 30 23:00:03.476078 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:00:03.475717    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890020-vktnq" event={"ID":"5fc370b8-182b-4f2d-9731-8810be2bac99","Type":"ContainerDied","Data":"eb4d228d607ed9cacc15fec134b78f595d10b6e6f38b3762b92c02ea711b500c"}
Oct 30 23:00:03.476078 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:00:03.475770    5678 scope.go:122] "RemoveContainer" containerID="5e1679c29f23e90574d2b098e2e96de5a0845f2724db17f75bfddd453084c6a1"
Oct 30 23:00:03.476078 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:00:03.475994    5678 scope.go:122] "RemoveContainer" containerID="eb4d228d607ed9cacc15fec134b78f595d10b6e6f38b3762b92c02ea711b500c"
Oct 30 23:00:03.476244 ip-10-0-128-245 kubenswrapper[5678]: E1030 23:00:03.476226    5678 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890020-vktnq_odh-ai-gateway-infra(5fc370b8-182b-4f2d-9731-8810be2bac99)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890020-vktnq" podUID="5fc370b8-182b-4f2d-9731-8810be2bac99"
Oct 30 23:00:04.483091 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:00:04.483065    5678 scope.go:122] "RemoveContainer" containerID="eb4d228d607ed9cacc15fec134b78f595d10b6e6f38b3762b92c02ea711b500c"
Oct 30 23:00:04.483524 ip-10-0-128-245 kubenswrapper[5678]: E1030 23:00:04.483293    5678 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890020-vktnq_odh-ai-gateway-infra(5fc370b8-182b-4f2d-9731-8810be2bac99)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890020-vktnq" podUID="5fc370b8-182b-4f2d-9731-8810be2bac99"
Oct 30 23:00:10.571896 ip-10-0-128-245 kubenswrapper[5678]: E1030 23:00:10.571859    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 23:00:15.516569 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:00:15.516535    5678 scope.go:122] "RemoveContainer" containerID="eb4d228d607ed9cacc15fec134b78f595d10b6e6f38b3762b92c02ea711b500c"
Oct 30 23:00:16.546713 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:00:16.546680    5678 generic.go:356] "Generic (PLEG): container finished" podID="5fc370b8-182b-4f2d-9731-8810be2bac99" containerID="a234cd3dcb424bfb5f995f61b6b784f3b88149c01254a3cce7c83961ec525b19" exitCode=60
Oct 30 23:00:16.547155 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:00:16.546751    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890020-vktnq" event={"ID":"5fc370b8-182b-4f2d-9731-8810be2bac99","Type":"ContainerDied","Data":"a234cd3dcb424bfb5f995f61b6b784f3b88149c01254a3cce7c83961ec525b19"}
Oct 30 23:00:16.547155 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:00:16.546817    5678 scope.go:122] "RemoveContainer" containerID="eb4d228d607ed9cacc15fec134b78f595d10b6e6f38b3762b92c02ea711b500c"
Oct 30 23:00:16.547155 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:00:16.547133    5678 scope.go:122] "RemoveContainer" containerID="a234cd3dcb424bfb5f995f61b6b784f3b88149c01254a3cce7c83961ec525b19"
Oct 30 23:00:16.547384 ip-10-0-128-245 kubenswrapper[5678]: E1030 23:00:16.547362    5678 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29890020-vktnq_odh-ai-gateway-infra(5fc370b8-182b-4f2d-9731-8810be2bac99)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890020-vktnq" podUID="5fc370b8-182b-4f2d-9731-8810be2bac99"
Oct 30 23:00:17.667863 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:00:17.667826    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890020-vktnq"]
Oct 30 23:00:17.806217 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:00:17.806192    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890020-vktnq"
Oct 30 23:00:17.852237 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:00:17.852205    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5fc370b8-182b-4f2d-9731-8810be2bac99-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fc370b8-182b-4f2d-9731-8810be2bac99-maas-api-service-ca\") pod \"5fc370b8-182b-4f2d-9731-8810be2bac99\" (UID: \"5fc370b8-182b-4f2d-9731-8810be2bac99\") "
Oct 30 23:00:17.852445 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:00:17.852285    5678 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fc370b8-182b-4f2d-9731-8810be2bac99-kube-api-access-nmr6r\" (UniqueName: \"kubernetes.io/projected/5fc370b8-182b-4f2d-9731-8810be2bac99-kube-api-access-nmr6r\") pod \"5fc370b8-182b-4f2d-9731-8810be2bac99\" (UID: \"5fc370b8-182b-4f2d-9731-8810be2bac99\") "
Oct 30 23:00:17.852669 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:00:17.852645    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fc370b8-182b-4f2d-9731-8810be2bac99-maas-api-service-ca" pod "5fc370b8-182b-4f2d-9731-8810be2bac99" (UID: "5fc370b8-182b-4f2d-9731-8810be2bac99"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:00:17.854428 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:00:17.854393    5678 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fc370b8-182b-4f2d-9731-8810be2bac99-kube-api-access-nmr6r" pod "5fc370b8-182b-4f2d-9731-8810be2bac99" (UID: "5fc370b8-182b-4f2d-9731-8810be2bac99"). InnerVolumeSpecName "kube-api-access-nmr6r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:00:17.954005 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:00:17.953929    5678 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fc370b8-182b-4f2d-9731-8810be2bac99-maas-api-service-ca\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:00:17.954005 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:00:17.953957    5678 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmr6r\" (UniqueName: \"kubernetes.io/projected/5fc370b8-182b-4f2d-9731-8810be2bac99-kube-api-access-nmr6r\") on node \"ip-10-0-128-245.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:00:18.561418 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:00:18.561366    5678 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890020-vktnq"
Oct 30 23:00:18.561587 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:00:18.561364    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890020-vktnq" event={"ID":"5fc370b8-182b-4f2d-9731-8810be2bac99","Type":"ContainerDied","Data":"396acd9b9cbf9eadd38744a828781dc906c7960e5044026bba64f92d326a1bd0"}
Oct 30 23:00:18.561587 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:00:18.561485    5678 scope.go:122] "RemoveContainer" containerID="a234cd3dcb424bfb5f995f61b6b784f3b88149c01254a3cce7c83961ec525b19"
Oct 30 23:00:18.751631 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:00:18.751604    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890020-vktnq"]
Oct 30 23:00:18.756357 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:00:18.756329    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890020-vktnq"]
Oct 30 23:00:20.522373 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:00:20.522343    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fc370b8-182b-4f2d-9731-8810be2bac99" path="/var/lib/kubelet/pods/5fc370b8-182b-4f2d-9731-8810be2bac99/volumes"
Oct 30 23:00:40.571924 ip-10-0-128-245 kubenswrapper[5678]: E1030 23:00:40.571889    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 23:01:10.573144 ip-10-0-128-245 kubenswrapper[5678]: E1030 23:01:10.573055    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 23:01:40.570820 ip-10-0-128-245 kubenswrapper[5678]: E1030 23:01:40.570784    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 23:01:47.072424 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:01:47.066305    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-mtqlj"]
Oct 30 23:01:47.107142 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:01:47.107108    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-mtqlj"]
Oct 30 23:01:48.522150 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:01:48.522115    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5006f08e-1ebc-4e9f-bad6-bac41a147461" path="/var/lib/kubelet/pods/5006f08e-1ebc-4e9f-bad6-bac41a147461/volumes"
Oct 30 23:02:10.571258 ip-10-0-128-245 kubenswrapper[5678]: E1030 23:02:10.571220    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 23:02:16.046051 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:02:16.046017    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-bmtdp"]
Oct 30 23:02:16.092819 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:02:16.092793    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-bmtdp"]
Oct 30 23:02:16.521363 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:02:16.521330    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56a5a108-7114-47cd-a79c-aa2723254d6d" path="/var/lib/kubelet/pods/56a5a108-7114-47cd-a79c-aa2723254d6d/volumes"
Oct 30 23:02:40.570365 ip-10-0-128-245 kubenswrapper[5678]: E1030 23:02:40.570282    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 23:02:50.057221 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:02:50.057185    5678 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5b97c83-njjk9"]
Oct 30 23:02:50.103756 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:02:50.103723    5678 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5b97c83-njjk9"]
Oct 30 23:02:50.523598 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:02:50.523564    5678 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50d35d93-4c69-4451-a0a8-82a9ee9740bd" path="/var/lib/kubelet/pods/50d35d93-4c69-4451-a0a8-82a9ee9740bd/volumes"
Oct 30 23:03:01.707563 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:03:01.707527    5678 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-755df48df5-dhkgj_87daad87-329d-4840-a2e8-61fd6f03beac/manager/0.log"
Oct 30 23:03:03.586674 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:03:03.586644    5678 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-8vwn4_057ad8c2-55df-4e8e-b599-e722ea8b9925/server/0.log"
Oct 30 23:03:04.924885 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:03:04.924857    5678 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-57df996b79-zkxjh_957947a4-ac1a-4dcd-aafa-e79848117b51/maas-api/0.log"
Oct 30 23:03:08.257890 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:03:08.257857    5678 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-w2tw2_fb169f7f-ab9f-4619-989c-20b7c5c150e9/manager/0.log"
Oct 30 23:03:08.578680 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:03:08.578609    5678 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-446sc_fcbb7898-1962-48be-83bb-0bc88f34674d/manager/0.log"
Oct 30 23:03:08.805766 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:03:08.805741    5678 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-vdfd2_257a2b42-2326-44b2-840b-a2e974882d20/kuadrant-console-plugin/0.log"
Oct 30 23:03:10.570698 ip-10-0-128-245 kubenswrapper[5678]: E1030 23:03:10.570665    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 23:03:11.251202 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:03:11.251164    5678 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-84877cfc-5ktml_b8e83b65-8951-43d2-8943-0a0bce44b40c/kube-auth-proxy/0.log"
Oct 30 23:03:11.701844 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:03:11.701815    5678 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-85544d4cd7-269mj_db21992e-098d-4659-966f-08b506d342dc/istio-proxy/0.log"
Oct 30 23:03:12.265273 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:03:12.265244    5678 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-85544d4cd7-8mdvr_c4729d87-68c3-4c11-9a1d-c235bc370104/istio-proxy/0.log"
Oct 30 23:03:14.175056 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:03:14.175027    5678 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-85544d4cd7-xvtp9_8194973b-fc01-458a-931f-562b4da099ae/istio-proxy/0.log"
Oct 30 23:03:16.318335 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:03:16.318303    5678 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-plz4v_c32955de-27bc-4ed0-83ab-7cf9f4a8b6cf/main/0.log"
Oct 30 23:03:16.764726 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:03:16.764695    5678 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-klr4q_0b10aa0a-17b4-427e-a9d8-b9202ba1ac0a/main/0.log"
Oct 30 23:03:17.672985 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:03:17.672952    5678 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-9ndd9_81bd6415-f42f-446b-a64e-956f794b3956/main/0.log"
Oct 30 23:03:24.820421 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:03:24.820373    5678 scope.go:122] "RemoveContainer" containerID="d00ffd404f3792798fbe72cab767c75d63922ee0d6ce551175bd3074781f8b70"
Oct 30 23:03:24.829073 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:03:24.829053    5678 scope.go:122] "RemoveContainer" containerID="b734b06787af7679f5b44003d86bcec8d39aed068ce1365787283fc06d4b3420"
Oct 30 23:03:28.139765 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:03:28.139736    5678 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-755df48df5-dhkgj_87daad87-329d-4840-a2e8-61fd6f03beac/manager/0.log"
Oct 30 23:03:29.788537 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:03:29.788514    5678 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-8vwn4_057ad8c2-55df-4e8e-b599-e722ea8b9925/server/0.log"
Oct 30 23:03:30.976941 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:03:30.976908    5678 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-57df996b79-zkxjh_957947a4-ac1a-4dcd-aafa-e79848117b51/maas-api/0.log"
Oct 30 23:03:33.846479 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:03:33.846447    5678 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-w2tw2_fb169f7f-ab9f-4619-989c-20b7c5c150e9/manager/0.log"
Oct 30 23:03:34.156396 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:03:34.156368    5678 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-446sc_fcbb7898-1962-48be-83bb-0bc88f34674d/manager/0.log"
Oct 30 23:03:34.365013 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:03:34.364987    5678 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-vdfd2_257a2b42-2326-44b2-840b-a2e974882d20/kuadrant-console-plugin/0.log"
Oct 30 23:03:36.244867 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:03:36.244832    5678 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-84877cfc-5ktml_b8e83b65-8951-43d2-8943-0a0bce44b40c/kube-auth-proxy/0.log"
Oct 30 23:03:36.705046 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:03:36.705018    5678 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-85544d4cd7-269mj_db21992e-098d-4659-966f-08b506d342dc/istio-proxy/0.log"
Oct 30 23:03:37.251865 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:03:37.251835    5678 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-85544d4cd7-8mdvr_c4729d87-68c3-4c11-9a1d-c235bc370104/istio-proxy/0.log"
Oct 30 23:03:39.109340 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:03:39.109287    5678 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-85544d4cd7-xvtp9_8194973b-fc01-458a-931f-562b4da099ae/istio-proxy/0.log"
Oct 30 23:03:40.570444 ip-10-0-128-245 kubenswrapper[5678]: E1030 23:03:40.570394    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 23:03:41.009155 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:03:41.009121    5678 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-plz4v_c32955de-27bc-4ed0-83ab-7cf9f4a8b6cf/main/0.log"
Oct 30 23:03:41.457983 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:03:41.457953    5678 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-klr4q_0b10aa0a-17b4-427e-a9d8-b9202ba1ac0a/main/0.log"
Oct 30 23:03:42.343670 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:03:42.343641    5678 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-9ndd9_81bd6415-f42f-446b-a64e-956f794b3956/main/0.log"
Oct 30 23:04:10.571556 ip-10-0-128-245 kubenswrapper[5678]: E1030 23:04:10.571476    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 23:04:24.852693 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:04:24.852664    5678 scope.go:122] "RemoveContainer" containerID="c548848b7ffe53cd6cdb2abdaeb759552b1b79d28b412b51ce8d40226685ccb9"
Oct 30 23:04:40.571148 ip-10-0-128-245 kubenswrapper[5678]: E1030 23:04:40.571106    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 23:05:05.211937 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:05:05.211908    5678 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-cck7d_b954bab2-9dd7-4b39-bda6-3fec87381aff/global-pull-secret-syncer/0.log"
Oct 30 23:05:08.645543 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:05:08.645507    5678 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-w72c9_0ad59239-edcc-46a4-a5a1-1debb63e3a5d/konnectivity-agent/0.log"
Oct 30 23:05:09.020833 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:05:09.020752    5678 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-245.us-west-2.compute.internal_bb28c8c054a6af006e24e10c141990f3/haproxy/0.log"
Oct 30 23:05:10.570608 ip-10-0-128-245 kubenswrapper[5678]: E1030 23:05:10.570577    5678 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kn6xx" containerName="extractor"
Oct 30 23:05:14.979318 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:05:14.979284    5678 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-w2tw2_fb169f7f-ab9f-4619-989c-20b7c5c150e9/manager/0.log"
Oct 30 23:05:14.992698 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:05:14.992669    5678 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-2cfz8/perf-node-gather-daemonset-nmt2x"]
Oct 30 23:05:14.993028 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:05:14.993016    5678 cpu_manager.go:414] "containerMap: removing container" podUID="5fc370b8-182b-4f2d-9731-8810be2bac99" containerName="cleanup"
Oct 30 23:05:14.993076 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:05:14.993029    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fc370b8-182b-4f2d-9731-8810be2bac99" containerName="cleanup"
Oct 30 23:05:14.993076 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:05:14.993051    5678 cpu_manager.go:414] "containerMap: removing container" podUID="5fc370b8-182b-4f2d-9731-8810be2bac99" containerName="cleanup"
Oct 30 23:05:14.993076 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:05:14.993056    5678 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fc370b8-182b-4f2d-9731-8810be2bac99" containerName="cleanup"
Oct 30 23:05:14.993194 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:05:14.993127    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fc370b8-182b-4f2d-9731-8810be2bac99" containerName="cleanup"
Oct 30 23:05:14.993194 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:05:14.993141    5678 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fc370b8-182b-4f2d-9731-8810be2bac99" containerName="cleanup"
Oct 30 23:05:14.996361 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:05:14.996345    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2cfz8/perf-node-gather-daemonset-nmt2x"
Oct 30 23:05:15.048570 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:05:15.048547    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-2cfz8\"/\"kube-root-ca.crt\""
Oct 30 23:05:15.094762 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:05:15.094733    5678 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-2cfz8\"/\"openshift-service-ca.crt\""
Oct 30 23:05:15.094935 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:05:15.094926    5678 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-2cfz8\"/\"default-dockercfg-4lpsw\""
Oct 30 23:05:15.096936 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:05:15.096910    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2cfz8/perf-node-gather-daemonset-nmt2x"]
Oct 30 23:05:15.103240 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:05:15.103210    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a4648f04-e500-4c7a-bc3e-0bbcfb664676-podres\") pod \"perf-node-gather-daemonset-nmt2x\" (UID: \"a4648f04-e500-4c7a-bc3e-0bbcfb664676\") " pod="openshift-must-gather-2cfz8/perf-node-gather-daemonset-nmt2x"
Oct 30 23:05:15.103368 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:05:15.103253    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a4648f04-e500-4c7a-bc3e-0bbcfb664676-proc\") pod \"perf-node-gather-daemonset-nmt2x\" (UID: \"a4648f04-e500-4c7a-bc3e-0bbcfb664676\") " pod="openshift-must-gather-2cfz8/perf-node-gather-daemonset-nmt2x"
Oct 30 23:05:15.103368 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:05:15.103284    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdlkf\" (UniqueName: \"kubernetes.io/projected/a4648f04-e500-4c7a-bc3e-0bbcfb664676-kube-api-access-vdlkf\") pod \"perf-node-gather-daemonset-nmt2x\" (UID: \"a4648f04-e500-4c7a-bc3e-0bbcfb664676\") " pod="openshift-must-gather-2cfz8/perf-node-gather-daemonset-nmt2x"
Oct 30 23:05:15.103368 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:05:15.103355    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a4648f04-e500-4c7a-bc3e-0bbcfb664676-sys\") pod \"perf-node-gather-daemonset-nmt2x\" (UID: \"a4648f04-e500-4c7a-bc3e-0bbcfb664676\") " pod="openshift-must-gather-2cfz8/perf-node-gather-daemonset-nmt2x"
Oct 30 23:05:15.103595 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:05:15.103450    5678 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a4648f04-e500-4c7a-bc3e-0bbcfb664676-lib-modules\") pod \"perf-node-gather-daemonset-nmt2x\" (UID: \"a4648f04-e500-4c7a-bc3e-0bbcfb664676\") " pod="openshift-must-gather-2cfz8/perf-node-gather-daemonset-nmt2x"
Oct 30 23:05:15.203976 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:05:15.203944    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a4648f04-e500-4c7a-bc3e-0bbcfb664676-podres\") pod \"perf-node-gather-daemonset-nmt2x\" (UID: \"a4648f04-e500-4c7a-bc3e-0bbcfb664676\") " pod="openshift-must-gather-2cfz8/perf-node-gather-daemonset-nmt2x"
Oct 30 23:05:15.204167 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:05:15.204002    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a4648f04-e500-4c7a-bc3e-0bbcfb664676-proc\") pod \"perf-node-gather-daemonset-nmt2x\" (UID: \"a4648f04-e500-4c7a-bc3e-0bbcfb664676\") " pod="openshift-must-gather-2cfz8/perf-node-gather-daemonset-nmt2x"
Oct 30 23:05:15.204167 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:05:15.204033    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdlkf\" (UniqueName: \"kubernetes.io/projected/a4648f04-e500-4c7a-bc3e-0bbcfb664676-kube-api-access-vdlkf\") pod \"perf-node-gather-daemonset-nmt2x\" (UID: \"a4648f04-e500-4c7a-bc3e-0bbcfb664676\") " pod="openshift-must-gather-2cfz8/perf-node-gather-daemonset-nmt2x"
Oct 30 23:05:15.204167 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:05:15.204057    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a4648f04-e500-4c7a-bc3e-0bbcfb664676-sys\") pod \"perf-node-gather-daemonset-nmt2x\" (UID: \"a4648f04-e500-4c7a-bc3e-0bbcfb664676\") " pod="openshift-must-gather-2cfz8/perf-node-gather-daemonset-nmt2x"
Oct 30 23:05:15.204167 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:05:15.204071    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a4648f04-e500-4c7a-bc3e-0bbcfb664676-podres\") pod \"perf-node-gather-daemonset-nmt2x\" (UID: \"a4648f04-e500-4c7a-bc3e-0bbcfb664676\") " pod="openshift-must-gather-2cfz8/perf-node-gather-daemonset-nmt2x"
Oct 30 23:05:15.204167 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:05:15.204102    5678 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a4648f04-e500-4c7a-bc3e-0bbcfb664676-lib-modules\") pod \"perf-node-gather-daemonset-nmt2x\" (UID: \"a4648f04-e500-4c7a-bc3e-0bbcfb664676\") " pod="openshift-must-gather-2cfz8/perf-node-gather-daemonset-nmt2x"
Oct 30 23:05:15.204167 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:05:15.204115    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a4648f04-e500-4c7a-bc3e-0bbcfb664676-proc\") pod \"perf-node-gather-daemonset-nmt2x\" (UID: \"a4648f04-e500-4c7a-bc3e-0bbcfb664676\") " pod="openshift-must-gather-2cfz8/perf-node-gather-daemonset-nmt2x"
Oct 30 23:05:15.204167 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:05:15.204155    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a4648f04-e500-4c7a-bc3e-0bbcfb664676-sys\") pod \"perf-node-gather-daemonset-nmt2x\" (UID: \"a4648f04-e500-4c7a-bc3e-0bbcfb664676\") " pod="openshift-must-gather-2cfz8/perf-node-gather-daemonset-nmt2x"
Oct 30 23:05:15.204470 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:05:15.204184    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a4648f04-e500-4c7a-bc3e-0bbcfb664676-lib-modules\") pod \"perf-node-gather-daemonset-nmt2x\" (UID: \"a4648f04-e500-4c7a-bc3e-0bbcfb664676\") " pod="openshift-must-gather-2cfz8/perf-node-gather-daemonset-nmt2x"
Oct 30 23:05:15.263004 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:05:15.262930    5678 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdlkf\" (UniqueName: \"kubernetes.io/projected/a4648f04-e500-4c7a-bc3e-0bbcfb664676-kube-api-access-vdlkf\") pod \"perf-node-gather-daemonset-nmt2x\" (UID: \"a4648f04-e500-4c7a-bc3e-0bbcfb664676\") " pod="openshift-must-gather-2cfz8/perf-node-gather-daemonset-nmt2x"
Oct 30 23:05:15.307996 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:05:15.307968    5678 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2cfz8/perf-node-gather-daemonset-nmt2x"
Oct 30 23:05:15.381144 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:05:15.381097    5678 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-446sc_fcbb7898-1962-48be-83bb-0bc88f34674d/manager/0.log"
Oct 30 23:05:15.687103 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:05:15.687060    5678 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2cfz8/perf-node-gather-daemonset-nmt2x"]
Oct 30 23:05:15.688795 ip-10-0-128-245 kubenswrapper[5678]: W1030 23:05:15.688769    5678 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poda4648f04_e500_4c7a_bc3e_0bbcfb664676.slice/crio-fe0ea23d8bdce9981d397e7655b754d7b32351570351a2684442057105bcd4a3 WatchSource:0}: Error finding container fe0ea23d8bdce9981d397e7655b754d7b32351570351a2684442057105bcd4a3: Status 404 returned error can't find the container with id fe0ea23d8bdce9981d397e7655b754d7b32351570351a2684442057105bcd4a3
Oct 30 23:05:15.779734 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:05:15.779710    5678 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-vdfd2_257a2b42-2326-44b2-840b-a2e974882d20/kuadrant-console-plugin/0.log"
Oct 30 23:05:16.071297 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:05:16.071206    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2cfz8/perf-node-gather-daemonset-nmt2x" event={"ID":"a4648f04-e500-4c7a-bc3e-0bbcfb664676","Type":"ContainerStarted","Data":"807f13e92cfb786b52306d6051ac734dd5f7ba9f04731a6a9128836ca1d8b877"}
Oct 30 23:05:16.071297 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:05:16.071260    5678 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2cfz8/perf-node-gather-daemonset-nmt2x" event={"ID":"a4648f04-e500-4c7a-bc3e-0bbcfb664676","Type":"ContainerStarted","Data":"fe0ea23d8bdce9981d397e7655b754d7b32351570351a2684442057105bcd4a3"}
Oct 30 23:05:16.071873 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:05:16.071305    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-2cfz8/perf-node-gather-daemonset-nmt2x"
Oct 30 23:05:16.181433 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:05:16.181369    5678 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-2cfz8/perf-node-gather-daemonset-nmt2x" podStartSLOduration=2.1813559319999998 podStartE2EDuration="2.181355932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:05:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:05:16.180770229 +0000 UTC m=+4493.785077779" watchObservedRunningTime="2026-10-30 23:05:16.181355932 +0000 UTC m=+4493.785663484"
Oct 30 23:05:22.087243 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:05:22.087216    5678 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-2cfz8/perf-node-gather-daemonset-nmt2x"
Oct 30 23:05:22.990738 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:05:22.990706    5678 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-pvl2l_dec1cd54-fec6-4c12-9d37-a1b0b1a7ffc0/kube-state-metrics/0.log"
Oct 30 23:05:23.385839 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:05:23.385793    5678 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-pvl2l_dec1cd54-fec6-4c12-9d37-a1b0b1a7ffc0/kube-rbac-proxy-main/0.log"
Oct 30 23:05:23.785969 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:05:23.785892    5678 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-pvl2l_dec1cd54-fec6-4c12-9d37-a1b0b1a7ffc0/kube-rbac-proxy-self/0.log"
Oct 30 23:05:24.187179 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:05:24.187149    5678 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-dc7c7d47d-ghdn7_a3459ebb-2fc2-4a82-b9c0-de5542d667e1/metrics-server/0.log"
Oct 30 23:05:25.009833 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:05:25.009804    5678 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7j5j6_f938a93b-29dd-4e7d-b8e9-444df68acf03/node-exporter/0.log"
Oct 30 23:05:25.420672 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:05:25.420621    5678 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7j5j6_f938a93b-29dd-4e7d-b8e9-444df68acf03/kube-rbac-proxy/0.log"
Oct 30 23:05:25.860062 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:05:25.859987    5678 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7j5j6_f938a93b-29dd-4e7d-b8e9-444df68acf03/init-textfile/0.log"
Oct 30 23:05:29.943653 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:05:29.943626    5678 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_11d752aa-0037-43c4-aadc-1bd78e14918e/prometheus/0.log"
Oct 30 23:05:30.339895 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:05:30.339825    5678 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_11d752aa-0037-43c4-aadc-1bd78e14918e/config-reloader/0.log"
Oct 30 23:05:30.738118 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:05:30.738089    5678 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_11d752aa-0037-43c4-aadc-1bd78e14918e/thanos-sidecar/0.log"
Oct 30 23:05:31.136027 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:05:31.135957    5678 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_11d752aa-0037-43c4-aadc-1bd78e14918e/kube-rbac-proxy-web/0.log"
Oct 30 23:05:31.535612 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:05:31.535539    5678 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_11d752aa-0037-43c4-aadc-1bd78e14918e/kube-rbac-proxy/0.log"
Oct 30 23:05:31.937589 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:05:31.937565    5678 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_11d752aa-0037-43c4-aadc-1bd78e14918e/kube-rbac-proxy-thanos/0.log"
Oct 30 23:05:32.355620 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:05:32.355547    5678 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_11d752aa-0037-43c4-aadc-1bd78e14918e/init-config-reloader/0.log"
Oct 30 23:05:32.759906 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:05:32.759874    5678 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-vcj4q_67b50e2a-4d01-4d5f-a5d1-7e80ee62642a/prometheus-operator/0.log"
Oct 30 23:05:33.194012 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:05:33.193983    5678 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-vcj4q_67b50e2a-4d01-4d5f-a5d1-7e80ee62642a/kube-rbac-proxy/0.log"
Oct 30 23:05:38.146960 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:05:38.146916    5678 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-84877cfc-5ktml_b8e83b65-8951-43d2-8943-0a0bce44b40c/kube-auth-proxy/0.log"
Oct 30 23:05:38.565939 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:05:38.565831    5678 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-85544d4cd7-269mj_db21992e-098d-4659-966f-08b506d342dc/istio-proxy/0.log"
Oct 30 23:05:38.979636 ip-10-0-128-245 kubenswrapper[5678]: I1030 23:05:38.979604    5678 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-85544d4cd7-8mdvr_c4729d87-68c3-4c11-9a1d-c235bc370104/istio-proxy/0.log"